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Executive Summary 


The Regional Educational Laboratories (RELs) are a networked system of 10 organizations that 
serve the educational needs of designated regions across the United States and its territories. The 
U.S. Department of Education (ED) is authorized by the Education Sciences Reform Act (ESRA) to 
award contracts to 10 RELs to support applied research, development, wide dissemination, and 
technical assistance activities." The REL program is administered by the Knowledge Utilization 
Division of the National Center for Education Evaluation and Regional Assistance (NCEERA) 
within ED’s Institute of Education Sciences (IES), which was established by ESRA in 2002. 

ESRA requires NCEERA to provide for independent evaluations of each of the RELs in carrying 
out their duties, and transmit these results to Congress, the National Board for Education Sciences, 
and the appropriate REL governing boards/ 1 In 2009, the Evaluation Division of the NCEERA 
contracted with Westat to conduct these evaluations of the 10 RELs that had 5-year contracts 
between 2006 and 2011, as well as an evaluation of the REL program as a whole. The evaluation 
addresses the following questions: 

■ What activities did the RELs undertake to fulfill the mission specified in ESRA? 

■ What were the technical quality and relevance of REL Fast Response Project reports 
published by IES and of the corresponding proposals? 

■ What were the technical quality and relevance of REL impact study reports published 
by IES and of the corresponding proposals? 

■ How relevant and useful were REL technical assistance products to the needs of states 
and districts in their regions? 

An interim report from the evaluation was released in September 2013, addressing the first two 
evaluation questions listed above. 4 This final report addresses the remaining two questions for the 
evaluation, relying on expert panel review of REL impact study reports published by September 1 , 
2011, and impact study proposals submitted by that date, and on a survey of state education agency 
and local school district administrators conducted between the October 2011 and May 2012. 


2 The portion of ESRA pertaining to the Regional Educational Laboratories, Section 174 of P.L. 107-279, is available at 
http:/ /ies.ed.rov/pdf/PL107-279.pdf [accessed April 19, 2013]. 

3 The evaluation requirement is specified in Section 1740 of ESRA. 

4 Carlson et al. 2013, available at http:/ /ies.ed.gov/ncee/pubs/20134014 . 
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Research Questions and Key Findings 5 

What were the technical quality and relevance of REL impact study reports published by 
IES and of the corresponding proposals? 

■ Expert panelists rated 8 IES-published impact study reports as, on average, between 
“strong” and “very strong” in quality (4.10 on a 5-point scale, with 5 being the highest 
value). They rated 1 1 selected impact study proposals, including the proposals for the 8 
impact studies, as, on average, between “adequate” and “strong” in quality (3.59 on a 5- 
point scale). 

■ Expert panelists rated the 8 IES-published reports as, on average, between “relevant” 
and “very relevant” (4.06 on a 5-point scale, with 5 being the highest value). They rated 
the 1 1 impact study proposals as, on average, between “adequate” in relevance and 
“relevant” (3.61 on a 5-point scale). 


How relevant and useful were the REL technical assistance products to the needs of the 
states and districts in their regions? 

What needs did state and district administrators have for education research and technical assistance, 
and were those needs met? 

■ The most commonly reported area of “high need” for education research and/ or 
technical assistance (as opposed to “moderate need” or “low or no need”) among state 
administrators was teacher/ staff evaluation (53 percent). The most commonly reported 
area of “high need” for education research and/ or technical assistance among district 
administrators was content standards, curriculum, or instruction in science, technology, 
engineering, and mathematics (STEM) (37 percent). 

■ Twenty-nine percent of state administrators and 26 percent of district administrators 
reported that their research and technical assistance needs were met “very well,” 
regardless of the source of that assistance. 

What sources of education research and technical assistance did state and district 
administrators use? 


■ The most commonly reported sources for education research and/ or technical 

assistance for state administrators were professional associations (87 percent, versus 49 
percent relying on the REL program). The most commonly reported sources for 
education research and/ or technical assistance for district administrators were their 


5 More detailed national findings from the expert panel review and survey of state and district administrators are 
provided in Chapter 3. 
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counterparts at other LEAs or SEAs (82 percent, versus 1 8 percent relying on the REL 
program). 

How familiar were state and district administrators with the REL program? 

■ Eighty-six percent of state administrators reported being “very familiar,” “somewhat 
familiar,” or “a little familiar” with the REL program, and 14 percent reported that they 
were “not familiar at all” with the REL program. In contrast, 52 percent of district 
administrators reported being “very familiar,” “somewhat familiar,” or “a little familiar” 
with the REL program, and 48 percent reported that they were “not familiar at all” with 
the REL program. 

How many state and district administrators used REL services? 

■ Seventy-seven percent of state administrators and 46 percent of district administrators 
who were at least “a little familiar” 6 with the REL program reported that they used one 
or more REL services in the past 12 months. 

How satisfied with the REL program were state and district administrators? 

■ One-half (50 percent) of state administrators and 26 percent of district administrators 
who were at least “a little familiar” with the REL program were “very satisfied” with it, 
48 percent of these state administrators and 58 percent of these district administrators 
were “somewhat satisfied” with the REL program, and 3 percent of state administrators 
and 16 percent of district administrators were “not at all satisfied” with the REL 
program. 

The sections below provide more background on the REL program and on the data collection and 
analysis conducted by the study team to answer the research questions for the final evaluation 
report. It is followed by a more extensive discussion of the evaluation findings. 


Background on the REL Program 

ED, through the Knowledge Utilization Division in NCEERA within IES, awarded 5-year contracts 
to ten RELs in FY 2006. These contracts were subsequently extended to December 31, 2011. Table 
ES-1 lists the states and territories in each region served by a Regional Educational Laboratory, as 
well as the organizations that held the REL contracts from 2006 to 2011. Annual appropriations for 
the REL program varied over the period of performance of the 2006-201 1 REL contracts. 


6 Unless otherwise specified, the term ‘ at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” and “a little familiar.” 
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Appropriations for the REL program in FY 2006 and under the FY 2007 continuing resolution 
equaled $65,470 million each year. Over the following three years, appropriations rose to $65,569 
million in FY 2008, $67,569 million in FY 2009, and $70,650 million in FY 2010. For FY 201 1 and 
FY 2012, appropriations fell to $57,535 and $57,426 million, respectively. 

Table ES-1. REL regions and prime contractors, 2006 - 2011 


Region 

States and territories 

Prime contractor, 2006-2011 

Appalachia (AP) 

Kentucky, Tennessee, Virginia, West Virginia 

CNA 

Central (CE) 

Colorado, Kansas, Missouri, Nebraska, North 
Dakota, South Dakota, Wyoming 

Mid-Continent Research for 
Education and Learning (McREL) 

Mid-Atlantic (MA) 

Delaware, Maryland, New Jersey, Pennsylvania, 
Washington, D.C. 

The Pennsylvania State University 
(PSU), with 4 primary 
subcontractors: Rutgers University, 
ICF International, ANALYTICA, and 
the Metiri Group 

Midwest (MW) 

Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, 
Wisconsin 

Learning Point Associates (LPA) 

Northeast & 
Islands (NE& 1) 

Connecticut, Maine, Massachusetts, New 
Hampshire, New York, Puerto Rico, Rhode Island, 
Vermont, Virgin Islands 

Education Development Center 
(EDC), with Learning Innovations 
at WestEd and the American 
Institutes for Research (AIR) as 
primary subcontractors. 

Northwest (NW) 

Alaska, Idaho, Montana, Oregon, Washington 

Education Northwest 

Pacific (PA) 

American Samoa, Federated States of Micronesia, 
Guam, Hawaii, Northern Mariana Islands, Republic 
of the Marshall Islands, Republic of Palau 

Pacific Resources for Education 
and Learning (PREL) 

Southeast (SE) 

Alabama, Florida, Georgia, Mississippi, North 
Carolina, South Carolina 

SERVE Center, University of North 
Carolina at Greensboro 

Southwest (SW) 

Arkansas, Louisiana, New Mexico, Oklahoma, 
Texas 

Edvance Research, Inc. 

West (W) 

Arizona, California, Nevada, Utah 

WestEd 


NOTE: Complete subcontracting arrangements are not shown and may have varied over time, even if the prime contractor remained the 
same. The end date for the 2006-2011 REL contracts was December 31, 2011. 


Under the contracts in place between 2006 and 2011, the RELs performed two broad categories of 
applied research projects to prepare reports that were released and disseminated as IES publications. 
“Fast Response Projects” (FRPs) were short-term education research and/or technical assistance 
projects intended to (a) respond to regional and national education needs and priorities and (b) 
inform policy and practice. FRPs used various methods, such as literature reviews, analyses of extant 
data, and qualitative studies. “Impact studies” were projects designed specifically to make causal 
inferences about an intervention, policy, or practice, typically using randomized controlled trials 
(RCTs). 
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What were the technical quality and relevance of REL impact study reports published by 
IES and of the corresponding proposals? 

The study team’s plan to evaluate the quality and relevance of REL impact study proposals and 
reports included use of an independent expert panel to review the technical quality and relevance of 
impact study reports released by IES and of the proposals that had resulted in those reports. 
Because of resource limitations for the evaluation, the evaluation study team and NCEE focused 
panel reviews on the beginning and end stages of IES-published reports. Revised proposals, 
proposals that were rejected or were still under review as of September 1, 2011, and proposals for 
reports that were not published by IES were all excluded from panel review. 

The evaluation study team collected from REL web sites and from the REL program office all of 
the REL impact study reports published by IES on its website, as well as the corresponding 
proposals that were produced by the ten RELs between March 1 , 2006 and September 1 , 2011. A 
total of eight impact studies had been completed; they resulted in eight published reports by seven 
RELs over this period of time. In order to ensure that all 10 RELs had impact-related documents 
for review, researchers also collected one initial proposal for each of the three RELs that did not 
have a published report but whose final report was close to publication. This resulted in 8 reports 
and 1 1 initial proposals for the expert panel review. All of the studies reviewed used randomized 
controlled trials. 

The study team developed two rubrics for use in the expert panel review: one for reports and one 
for proposals (Appendix A). The rubrics included two dimensions: technical quality and relevance. 
Each dimension was further defined by multiple indicators. The rubrics for rating proposals and 
reports shared 8 indicators of quality and 5 indicators of relevance. Seven additional indicators of 
quality and one additional indicator of relevance were included in the mbric for rating reports. One 
quality indicator was unique to the mbric for rating proposals. Indicators for the dimensions were 
rated on a 5-point scale, with 1 being the lowest rating and 5 being the highest rating. The rubrics 
gave quality ratings the adjectival labels of “very weak” (1), “weak” (2), “adequate (3), “strong” (4), 
or “very strong” (5). Relevance ratings received the adjectival labels of “not relevant” (1), 
“marginally relevant” (2), “adequate” (3), “relevant” (4), or “very relevant” (5). 


7 Under the 2006-2011 REL contracts, the RELs completed a total of 23 impact studies, which resulted in 24 impact 
study reports (two reports were published about one study). This included eight reports about the eight completed 
impact studies that were published before September 1, 2011 and were part of the review conducted for this 
evaluation. There were 15 impact studies completed and 16 reports published after September 1, 2011 that were not 
part of the review for this evaluation. 
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A team of 11 expert panelists 8 rated 8 IES-published impact study reports as, on average, 
between “strong” and “very strong” in quality (mean quality rating of 4.10 on a 5-point scale, 
with 5 being the highest value). The 1 1 impact study proposals selected for review, including 
proposals for the 8 studies producing IES-published reports, received a mean quality rating of 3.59, 
which is between “adequate” and “strong” on a 5-point scale. 

The expert panelists rated 8 IES-published reports as, on average, between “relevant” and 
“very relevant” (mean relevance rating of 4.06 on a 5-point scale, with 5 being the highest value). 
The 1 1 impact study proposals selected for review received a mean relevance rating of 3.61, which is 
between “adequate” in relevance and “relevant.” 

The body of the report provides more detailed REL-specific findings on the quality and relevance of 
the impact study reports and proposals. 

How relevant and useful were the REL technical assistance products to the needs of the 
states and districts in their regions? 

To evaluate the relevance and usefulness of REL research and technical assistance products to the 
needs of state and district administrators nationwide and within their regions, between October 2011 
and May 2012, the evaluation team conducted a survey of a nationally representative sample of 346 
state administrators and 4,834 district administrators. 9 The survey addressed respondents’ needs for 
education research and technical assistance, familiarity with and use of REL products and services, 
and satisfaction with the work of the REL program. 

Data collection for the web-based survey began in October 2011. Email and telephone follow-up 
was used with those who had not responded, and members of the evaluation team also offered to 
complete the survey with the administrators over the phone. A third of all the completed surveys 
were conducted over the phone, with the phone interviewer entering the administrators’ responses 
into the web survey. The evaluation team continued to contact administrators until data collection 
ended in early May 2012. The overall response rate was 87 percent for the state sample of 
administrators, and 80 percent for the district sample, resulting in data from 290 state administrator 
respondents and 3,709 district administrator respondents. 


8 Three of the 1 1 experts were assigned to review each product according to their content or methodology expertise. 

9 Based on input from a Technical Working Group, the evaluation team identified state and district administrators as 
the primary audience or customers for REL products and services. 
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What needs did state and district administrators have for education research and technical 
assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators was teacher/staff evaluation. The most commonly 
reported area of “high need” for education research and/or technical assistance among 
district administrators was content standards, curriculum, or instruction in STEM. 

■ Administrators were asked to indicate whether they had a “high need,” “moderate 
need,” or “low or no need” for research and/ or technical assistance in specific topic 
areas. The areas in which the five largest percentages of state administrators indicated a 
“high need” were teacher/ staff evaluation (53 percent), achievement gaps (50 percent), 
college or career readiness (49 percent), support for low-achieving schools (49 percent), 
and using data for decisions (47 percent). 

■ The areas in which the five largest percentages of district administrators indicated a 
“high need” for research and/ or technical assistance were content standards, curriculum 
or instruction in STEM (37 percent); using data for decisions (35 percent); achievement 
gaps (35 percent); content standards, curriculum or instruction in reading/writing (32 
percent); and assessment (31 percent). 

Twenty-nine percent of state administrators and 26 percent of district administrators 
reported that their research and technical assistance needs were met “very well” (as 
opposed to “moderately well” or “not well”), regardless of the source of assistance. 

■ The percentage of state administrators who reported that their needs were met “very 
well,” regardless of the source of assistance, varied by region, from a high of 60 percent 
in the Pacific to a low of 1 8 percent in the Midwest. The percentage of district 
administrators who reported that their needs were met “very well,” regardless of the 
source of assistance, also varied by region, from a high of 30 percent in the Southeast to 
a low of 18 percent in the Northwest. 


What sources of education research and technical assistance did state and district 
administrators use? 

The most commonly reported sources for education research and/ or technical assistance for 
state administrators were professional associations. The most commonly reported sources 
for education research and/or technical assistance for district administrators were 
counterparts at other LEAs or SEAs. 

■ State and district administrators reported that they used a variety of sources for meeting 
their research and/ or technical assistance needs. Professional associations and 
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counterparts in other states and districts were most commonly reported as being relied 
upon “to a great extent” or “to a moderate extent” as opposed to a “small extent” or 
not at all (States: 87 and 78 percent, respectively; Districts: 70 and 82 percent, 
respectively) (Figure ES-1). 

■ Almost half of state administrators (49 percent) and 1 8 percent of district administrators 
reported that they relied on the REL program “to a great extent” or “to moderate 
extent” for research and/or technical assistance (Figure ES-1). 

■ State administrators’ reliance on the REL program varied across the regions: the 
percentage who reported relying on the REL program for research and/ or technical 
assistance “to a great extent” or “to a moderate extent” ranged from a high of 71 
percent in the West to a low of 27 percent in the Southwest. For district administrators, 
the percentage who reported relying on the REL program “to a great extent” or “to a 
moderate extent” ranged from a high of 22 percent in the Northwest to a low of 15 
percent in the Midwest. 

■ Eighty-eight percent of state administrators and 93 percent of district administrators 
reported that it was “very easy” or “moderately easy” (as opposed to “not at all easy”) to 
access education research and/ or technical assistance across the available sources of 
information. 

How familiar were state and district administrators with the REL program? 

Eighty-six percent of state administrators reported being at least “ a little familiar” with the 
REL program. In contrast, 52 percent of district administrators reported being at least “a 
little familiar” with the REL program. 

■ More than three-fourths (86 percent) of state administrators reported that they were 
“very familiar,” “somewhat familiar,” or “a little familiar” with the REL program, and 
more than half (52 percent) of district administrators reported that they were had at least 
“a little” familiarity with the program. 

■ Fourteen percent of state administrators reported that they were “not familiar at all” 
with the REL program, and almost one-half (48 percent) of district administrators 
reported that they were “not familiar at all” with it. 

In all ten regions, more than 70 percent of state administrators reported being atleast u a 
little familiar” with the REL program. 

■ In all 10 regions, more than 70 percent of state administrators reported being “very 
familiar,” “somewhat familiar,” or “a little familiar” with the REL program, ranging 
from a high of 96 percent in Northwest to a low of 71 percent in Southwest. REL 
Southwest was a first-time REL grantee in FY 2006. 
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Figure ES-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— National: School year 2011-12 


Sources of education research 
and technical assistance 

REL program nationwide 
Comprehensive centers 
Other federally funded providers 
Other products or resources from ED,. 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 


□ State 



% 


NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works." The total Ns for state and 
district administrators on the item about reliance on the REL program were 289 and 3,700, respectively. The total N for state 
administrators on the items about other specified sources of research ranged from 284 to 288, depending on the number of state 
respondents who chose not to respond to an individual item. The total N for district administrators on the items about other specified 
sources of research ranged from 3,684 to 3,694, depending on the number of district respondents who chose not to respond to an 
individual item. The total N for district administrators for “Other sources" was 217. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ The percentage of district administrators who reported that they were at least “a little 
familiar” with the REL program ranged from a high of 68 percent in Northwest to a 
low of 44 percent in Southwest. 


How many state and district administrators use REL services? 

Seventy-seven percent of state administrators and 46 percent of district administrators who were at 
least “a little familiar” with the REL program reported that they used one or more REL services in 
the past 12 months. Note that administrators’ use of services was contingent on familiarity, which 
differed considerably for states and districts. 

■ Of state administrators who were at least “a little familiar” with the REL program, the 
percentages that reported that they used each major type of REL service were 55 
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percent for technical assistance, 46 percent for a live or virtual event, 45 percent for 
responses to data or research requests, and 41 percent for information on the REL’s 
website. Of district administrators at least “a little familiar” with the REL program, 34 
percent reported obtaining information from the REL’s website; 20 percent reported 
attending a live or virtual event; 1 3 percent reported receiving a response to a data or 
research request; and 1 1 percent reported receiving technical assistance. 

■ Nineteen percent of state administrators and 33 percent of district administrators 
reported that they were at least “a little familiar” with the REL program but did not use 
any REL services in the past 12 months. When asked why they had not used any REL 
services in the past year, more than half of state and district administrators (54 and 56 
percent, respectively) at least “a little familiar” with the REL program but not using 
services in the past 1 2 months from the REL in their region reported that their needs 
were met elsewhere. In addition, 43 percent of these state administrators and 58 percent 
of these district administrators said they did not use REL services because they did not 
know what services were available. 

How satisfied with the REL program are state and district administrators? 

One-half (50 percent) of state administrators and 26 percent of district administrators who 
were at least “ a little familiar” with the REL program were “very satisfied” with it. 

■ Of the state administrators who reported being at least “a little familiar” with the REL 
program, 50 percent reported being “very satisfied” with it; 48 percent reported being 
“somewhat satisfied” with it; and 3 percent reported being “not at all satisfied” with it 
(Figure ES-2). 

■ Of the district administrators who reported being at least “a little familiar” with the REL 
program, 26 percent reported being “very satisfied” with it; 58 percent reported being 
“somewhat satisfied” with it; and 16 percent report being “not at all satisfied” with it 
(Figure ES-2). 

■ Across the regions, the percentage of state administrators who were at least “a little 
familiar” with the REL program and reported being “very satisfied” with it ranged from 
a high of 65 percent in the Southeast region to a low of 28 percent in the Mid-Atlantic 
region. 

■ The percentage of district administrators who were at least “a little familiar” with the 
REL program and reported being “very satisfied” with it ranged from a high of 34 
percent in the West region to a low of 17 percent in the Midwest region. 
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Figure ES-2. Percentage of administrators at least “a little familiar” with the REL program who 
were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” with it— National: 
School year 2011-12 
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NOTE: The total N for state administrators was 216, and the total N for district administrators was 1,619. Percentages may not sum to 
100 due to rounding. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 

The body of the report provides more detailed REL-specific findings from the survey of state and 
district administrators. 
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Introduction 


1 


The Regional Educational Laboratories (RELs) are a networked system of 10 organizations that 
serve the educational needs of 10 designated regions across the United States and its territories 
(Table 1-1). The U.S. Department of Education (ED) is authorized by the Education Sciences 
Reform Act (ESRA) to award contracts to 10 RELs to support applied research, development, wide 
dissemination, and/ or technical assistance (TA) activities. 10 The REL program is administered by the 
Knowledge Utilization Division of the National Center for Education Evaluation and Regional 
Assistance (NCEERA) within ED’s Institute of Education Sciences (IES), which was established by 
ESRA in 2002. 

ED, through the Knowledge Utilization Division of the NCEERA within IES, awarded 5-year 
contracts to 10 RELs on a competitive basis in FY 2006. These contracts were subsequently 
extended to end in FY 2012. Table 1-2 lists the organizations holding the 10 REL contracts from 
2006 to 201 1 as well as their history of REL funding (i.e., whether they have held REL 
grants/contracts in the past). 


Table 1-1. States and territories served by each REL 


Region 

States and territories 

Appalachia (AP) 

Kentucky, Tennessee, Virginia, West Virginia 

Central (CE) 

Colorado, Kansas, Missouri, Nebraska, North Dakota, South Dakota, Wyoming 

Mid-Atlantic (MA) 

Delaware, District of Columbia, Maryland, New Jersey, Pennsylvania 

Midwest (MW) 

Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, Wisconsin 

Northeast & Islands 
(NE&I) 

Connecticut, Maine, Massachusetts, New Hampshire, New York, Puerto Rico, Rhode 
Island, Vermont, Virgin Islands 

Northwest (NW) 

Alaska, Idaho, Montana, Oregon, Washington 

Pacific (PA) 

American Samoa, Federated States of Micronesia, Guam, Hawaii, Northern Mariana 
Islands, Republic of the Marshall Islands, Republic of Palau 

Southeast (SE) 

Alabama, Florida, Georgia, Mississippi, North Carolina, South Carolina 

Southwest (SW) 

Arkansas, Louisiana, New Mexico, Oklahoma, Texas 

West (W) 

Arizona, California, Nevada, Utah 


10 The portion of ESRA pertaining to the Regional Educational Laboratories, Section 174 of P.L. 107-279, is available at 
http:/ /ies. ed.gov/pdf/PL107-279.pdf [accessed April 19, 2013]. 
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Table 1-2. REL prime contractors, 2006-2011 


Region 

Prime contractor 

History of funding 

Appalachia 

CNA 

CNA received a REL contract for the first time 
in FY 2006 (awarded February 6, 2006). CNA 
was also awarded a REL contract for this 
region in FY 2012. 

Central 

Mid-Continent Research for Education 
and Learning (McREL) 

McREL had held the REL grant/contract 
continuously since 1966. The FY 2006 
contract was awarded on January 20, 2006. 
Marzano Research Laboratory was awarded a 
REL contract for this region in FY 2012. 

Mid-Atlantic 

The Pennsylvania State University 
(PSU), with 4 primary subcontractors: 
Rutgers University, ICF International, 
ANALYTICA, and the Metiri Group 

PSU received a REL contract for the first time 
in FY 2006 (awarded March 23, 2006). ICF 
International was awarded a REL contract for 
this region in FY 2012. 

Midwest 

Learning Point Associates (LPA) 

LPA had held a REL grant/contract since 
1984. The FY 2006 contract was awarded on 
March 9, 2006. The American Institutes for 
Research, (AIR) which merged with LPA in 
2011, was awarded a REL contract for this 
region in FY 2012. 

Northeast & 
Islands 

Education Development Center (EDC), 
with Learning Innovations at WestEd 
and AIR as primary subcontractors 

EDC held one of the original REL grants but did 
not hold one immediately before the FY 2006 
award. The FY 2006 contract was awarded on 
March 15, 2006. EDC was also awarded a REL 
contract for this region in FY 2012. 

Northwest 

Education Northwest 

Education Northwest, previously known as 
Northwest Regional Educational Laboratory, 
had held the REL grant/contract since 1966. 
The FY 2006 contract was awarded on 
February 1, 2006. Education Northwest was 
also awarded a REL contract for this region in 
FY 2012. 

Pacific 

Pacific Resources for Education and 
Learning (PREL) 

PREL had held the REL grant/contract since 
1990. The FY 2006 contract was awarded on 
March 16, 2006. McREL was awarded a REL 
contract for this region in FY 2012. 

Southeast 

SERVE Center, University of North 
Carolina at Greensboro 

SERVE had held the REL grant/contract since 
1990. The FY 2006 contract was awarded on 
March 16, 2006. Florida State University was 
awarded a REL contract for this region in FY 
2012. 

Southwest 

Edvance Research, Inc. 

Edvance received a REL contract for the first 
time in FY 2006 (awarded March 15, 2006). 
SEDL was awarded a REL contract for this 
region in FY 2012. 
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Table 1-2. REL prime contractors, 2006-2011 (continued) 


Region 

Prime contractor 

History of funding 

West 

WestEd 

WestEd had held the REL grant/contract since 
1966. The FY 2006 contract was awarded on 
January 18, 2006. WestEd was also awarded a 
REL contract for this region in FY 2012. 


NOTE: Complete subcontracting arrangements are not shown and may have varied over time, even if the prime contractor remained the 
same. The end date for the 2006-2011 REL contracts was December 31, 2011. 


Three contractors (CNA, Pennsylvania State University, and Edvance Research, Inc.) held first-time 
REL contracts in FY 2006; four contractors (Learning Point Associates, Education Development 
Center, Pacific Resources for Education and Learning, and SERVE Center at the University of 
North Carolina at Greensboro) held previous REL contracts; and three contractors (Mid-Continent 
Research for Education and Learning, Education Northwest, and WestEd) held continuous REL 
contracts since the inception of the program in 1966. Five of the 2006-11 prime contractors were 
awarded a REL contract for the same region in FY 2012. 

Annual appropriations for the REL program varied over the period of performance of the 2006- 
2011 REL contracts. Appropriations in FY 2006 and under the FY 2007 continuing resolution 
equaled $65,470 million each year. Over the following 3 years, appropriations rose, to $65,569 
million in FY 2008, $67,569 million in FY 2009, and $70,650 million in FY 2010. For FY 201 1 and 
FY 2012, appropriations fell to $57,535 and $57,426 million, respectively. 


Missions of the REL Program 


While the REL program was begun in 1966, it was most recently reauthorized under ESRA in 2002. 

Section 174(g) of ESRA specifies 10 missions for the RELs: 

1. Provide training and/ or technical assistance to constituents 

2. Disseminate scientifically valid research, information, reports, and publications that are 
usable for improving academic achievement, closing achievement gaps, and encouraging 
and sustaining school improvement 

3. Develop a plan for identifying and serving the needs of the region by conducting a 
continuing survey of the educational needs, strengths, and weaknesses within the region 
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4. Carry out applied research projects that are designed to serve the particular educational 
needs of the region, that reflect findings from scientifically valid research, and that result 
in user-friendly, replicable school-based classroom applications geared toward 
promoting student achievement 

5. Provide educational applied research in usable forms that promote school- 
improvement, academic achievement, and the closing of the achievement gaps and 
contribute to the current base of education knowledge by addressing problems in 
elementary and secondary education and access to postsecondary education 

6. Collaborate and coordinate services with other technical assistance providers funded by 
ED 

7. Assist in gathering information on school finance systems to promote improved access 
to educational opportunities and to better serve all public school students 

8. Assist in gathering information on alternative administrative structures that are more 
conducive to planning, implementing, and sustaining school reform and improved 
academic achievement 

9. Bring teams of experts together to develop and implement school improvement plans 
and strategies, especially in low-performing or high-poverty schools 

10. Develop innovative approaches to the application of technology in education that are 
unlikely to originate from within the private sector, but which could result in the 
development of new forms of education software, education content, and technology- 
enabled pedagogy 
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Table 1-3. Alignment of REL statement of work tasks with statutory missions for the RELs 


ESRA mission 

Task in the REL Statement of Work 

1. Provide training and/or technical assistance to 
constituents 

Regional education needs analysis, training 
and/or technical assistance, and fast- 
response applied research and 
development projects 

2. Disseminate scientifically valid research, information, 
reports, and publications that are usable for improving 
academic achievement, closing achievement gaps, and 
encouraging and sustaining school improvement 

Regional dissemination 

3. Develop a plan for identifying and serving the needs of the 
region by conducting a continuing survey of the 
educational needs, strengths, and weaknesses within the 
region 

Regional education needs analysis, training 
and/or technical assistance, and fast- 
response applied research and 
development projects 

4. Carry out applied research projects that are designed to 
serve the particular educational needs of the region, that 
reflect findings from scientifically valid research, and that 
result in user-friendly, replicable school-based classroom 
applications geared toward promoting student 
achievement 

Regional education needs analysis, training 
and assistance, and fast-response applied 
research and development projects 

5. Provide educational applied research in usable forms that 
promote school-improvement, academic achievement, 
and the closing of the achievement gaps and contribute to 
the current base of education knowledge by addressing 
problems in elementary and secondary education and 
access to postsecondary education 

Rigorous applied research and 
development 

6. Collaborate and coordinate services with other technical 
assistance providers funded by the Department of 
Education 

Coordination of REL network, website, and 
Intranet 

7. Assist in gathering information on school finance systems 
to promote improved access to educational opportunities 
and to better serve all public school students 

Not explicitly included in the REL statement 
of work 

8. Assist in gathering information on alternative 

administrative structures that are more conducive to 
planning, implementing, and sustaining school reform and 
improved academic achievement 

Not explicitly included in the REL statement 
of work 

9. Bring teams of experts together to develop and implement 
school improvement plans and strategies, especially in 
low-performing or high-poverty schools 

Not explicitly included in the REL statement 
of work 

10. Develop innovative approaches to the application of 

technology in education that are unlikely to originate from 
within the private sector, but which could result in the 
development of new forms of education software, 
education content, and technology-enabled pedagogy 

Not explicitly included in the REL statement 
of work 

Not explicitly included in the REL missions 

National Laboratory Network 

Not explicitly included in the REL missions 

Planning, management, and reporting 


Table Reads: The first statement of work task, Regional education needs analysis, training and/or technical assistance, and fast- 
response applied research and development projects, aligned with the first mission, Provide training and/or technical assistance to 
constituents. 

NOTE: A statement of work is a document developed by the government agency to define the activities, deliverables, and timeline 
required from an external source bidding on a procurement. 

SOURCE: Education Sciences Reform Act (P.L. 107-279) and the statement of work developed by NCEERA in 2005. 
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While most of the tasks of the REL statement of work (SOW) aligned explicitly with the missions 
specified for the RELs in ESRA, others did not (Table 1-3). The tasks that focused on regional 
needs identification, applied research and development projects, regional dissemination, and cross- 
REL coordination corresponded directly with the statutory missions for the REL program. The 
National Laboratory Network (NLN) task and the planning/management /reporting task did not 
correspond explicitly with any single REL mission specified in ESRA, but could be understood as 
supporting the performance of the other tasks. The NLN, for example, included Internet pages, 
accessible to the public from the IES website, describing each of the RELs and providing 
downloadable copies of IES-published REL reports. The NLN also included an Intranet for internal 
use in cross-REL collaboration and working groups. Four of the statutory missions of the REL 
program specified in ESRA — those focused on school finance, alternative administrative structures, 
school improvement strategies, and innovative technologies in education — were not explicit in the 
tasks of the SOW. 

Under the contracts in place between 2006 and 2011, the RELs performed two broad categories of 
projects to prepare reports that were released and disseminated as IES publications. “Fast Response 
Projects” (FRPs) were short-term education research and/or technical assistance projects intended 
to (1) respond to regional and national education needs and priorities and (2) inform policy and 
practice. FRPs used various methods, such as literature reviews, analyses of extant data, and 
qualitative studies. “Impact studies” were projects designed specifically to make causal inferences 
about an intervention, policy, or practice, typically using randomized controlled trials or regression 
discontinuity designs. Table 1-4 lists the number of IES- published FRP reports produced and IES- 
published impact studies conducted by each REL under their 2006-201 1 contract, as well as the total 
amount obligated for the contract period to each REL. Under the 2006-201 1 REL contracts, the 
RELs completed a total of 23 impact studies, which resulted in 24 impact study reports (two reports 
were published about one study). This included eight reports about the eight completed impact 
studies that were published before September 1, 201 1 and were part of the review conducted for this 
evaluation. There were 15 impact studies and 16 reports published after September 1, 201 1 that 
were not part of the review for this evaluation. 
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Table 1-4. Number of lES-published FRP reports, impact studies completed, and total amount 
obligated for the 2006-2011 contract, by REL 


REL 

Number of lES-published 
FRP reports 

Number of completed 
impact studies 

Total amount 
obligated 

Appalachia 

11 

1 

$28,481,655 

Central 

10 

3 

$27,651,454 

Mid-Atlantic 

12 

2 

$36,337,559 

Midwest 

15 

2 

$41,899,454 

Northeast & 
Islands 

23 

2 

$44,050,549 

Northwest 

15 

2 

$26,563,915 

Pacific 

7 

1 

$21,828,195 

Southeast 

12 

2 

$39,135,156 

Southwest 

22 

3 

$40,411,227 

West 

24 

6 

$45,383,654 


SOURCE: http://ies.ed. gov/pubsearch/index.asp?centername=REL&center=REL and http://govemment- 
contracts.findthebest.cem/guide . 


Independent Evaluation of the RELs 

Section 174(j) of ESRA requires NCEERA to provide for independent evaluations of each of the 
RELs in carrying out their duties, and transmit these results to Congress, the National Board for 
Education Sciences, and the appropriate REL governing boards. In 2009, the Evaluation Division of 
the NCEERA — which is administratively distinct from the Knowledge Utilization Division that 
manages the REL program — contracted with Westat to conduct these evaluations as well as an 
evaluation of the REL program as a whole. The REL program evaluation was designed to address, 
for the program and for each REL funded between 2006 and 2011, the following questions: 

■ What activities did the RELs undertake to fulfill the mission specified in ESRA? 

■ What were the technical quality and relevance of REL Fast Response Project reports 
published by IES and of the corresponding proposals? 

■ What were the technical quality and relevance of REL impact study reports published 
by IES and of the corresponding proposals? 

■ How relevant and useful were REL technical assistance products to the needs of states 
and districts in their regions? 
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An interim report from the evaluation was released in September 2013, addressing the first two 
evaluation questions listed above. 11 The results from the interim report are summarized briefly 
below as context for the findings from the final report. 

What Activities Did the RELs Undertake: Summary From the Interim Report 

To describe what activities the RELs had completed under the 10 missions specified in ESRA, 
including those not specified explicitly in the SOW, the evaluation study team conducted in-person 
interviews with staff members from each REL, including each REL’s director. Interviews with REL 
staff took place between May 2010 and July 2010 and lasted approximately 4 hours per REL. 
Interviewers from the study team asked the RELs to describe up to three major activities under each 
mission. The study team reviewed transcripts of the interviews to prepare a document that described 
the major REL activities reported by REL staff as addressing each of the 10 missions. To verify the 
accuracy of the write-ups, each REL director reviewed the draft document and made any necessary 
factual corrections. In general, the RELs reported activities under each of the statutory missions for 
the program. The documentation of REL activities as reported by REL staff is presented in REL- 
specific chapters of the interim report. 

What Were the Technical Quality and Relevance of REL Fast Response Project Reports 
Published by IES and of the Corresponding Proposals: Summary From the Interim Report 

The study team’s plan to evaluate the quality and relevance of REL Fast Response Project (FRP) 
reports and the corresponding proposals included (1) gathering data from the Knowledge Utilization 
Division of the NCEERA on the percentage of FRP proposals accepted for continuation as IES- 
supported studies, (2) gathering data from the Knowledge Utilization Division on the percentage of 
FRP reports accepted for publication as IES reports, and (3) having an independent expert panel 
review the technical quality and relevance of FRP reports released by IES and of the proposals that 
had resulted in those reports. 

As part of the FRP preparation and publication process, a REL would first submit a proposal for the 
FRP to its NCEERA project officer for review. Additional review of the proposal occurred through 
anonymous external reviewers working under the Analytical and Technical Support (ATS) contract 
that NCEERA awarded to Mathematica Policy Research, Inc. in 2006. The ATS reviews informed 
NCEERA’s decision to reject, approve, or request revisions to a REL proposal. For approved 
proposals, RELs would perform the work and submit draft reports to the NCEERA project officer 

11 Carlson et al. 2013, available at http:/ /ies. ed.gov/ncee /pubs /201 34014 . The Executive Summary of the interim report 
is reproduced as Chapter 2 of the final report. 
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for review. The ATS would again conduct an external review of the report. Also, at this point in the 
process the NCEERA Commissioner would transmit the report to the IES Standards and Review 
Office (SRO) for external review, resulting in revisions, and ultimately, the decision by the SRO 
office whether to approve the report for publication by IES. 

For the calculation of the percentage of FRP proposals and reports approved, the study team 
excluded “proposal or report reviews in process.” The denominator for the percentage of accepted 
FRP proposals included all of the FRP proposals that, by December 1 , 2009, were either accepted 
and authorized to proceed, or else rejected, but excluded proposals for which the NCEERA review 
was in process. The denominator for the percentage of accepted FRP reports included all of the 
reports that, by December 1, 2009, were either approved for publication or rejected, but excluded 
reports for which the IES review was in process. Readers should note that, even if a REL’s proposal 
for an FRP was accepted by NCEERA and the project was authorized to proceed, the resulting 
report could be rejected by NCEERA as a result of subsequent review by NCEERA or ATS 
contract staff. 

Of 297 proposals for FRPs reviewed by NCEERA by December 1, 2009, 46 percent (137) were 
accepted, and the REL was authorized to proceed with the project. The number of proposals 
submitted by each REL ranged from 17 to 45, and the percentage accepted by IES for each REL 
ranged from 24 to 67 percent. Of 166 FRP reports reviewed by IES by December 1, 2009, 55 
percent (92) were accepted for publication as IES reports. The number of reports submitted by each 
REL ranged from 1 0 to 27, and the percentage accepted ranged from 25 to 80 percent. 

In addition to calculating acceptance rates for FRP proposals and reports, the study team selected 
IES-published FRP reports and corresponding proposals for independent review by expert panels of 
methodological and content experts. Because of resource limitations for the evaluation, the 
evaluation study team and NCEERA focused panel reviews on the beginning and end of IES- 
published FRP reports. Revised proposals, proposals that were rejected or were still under review, 
and proposals for reports that were not published by IES were all excluded from panel review. 
Consequendy, findings on the quality and relevance of proposals refer only to proposals resulting in 
IES-published FRP reports that were released by December 1, 2009. The study team collected from 
REL web sites and from the REL program office all of the FRP reports that met these criteria, as 
well as the corresponding initial proposals that were submitted by the RELs. In total, the RELs 
produced 91 IES-published reports by this date, 88 of which were derived from 75 initial proposals. 
In some cases, a single proposal led to multiple IES-published reports (e.g., one for each state in a 
region). 
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The study team developed two rubrics for use in the FRP expert panel review: one for reports and 
one for initial proposals. The rubrics included two dimensions: technical quality and relevance. Each 
dimension was further defined by multiple indicators, which were rated on a 5-point scale, with 1 
being the lowest rating and 5 being the highest rating (Figure 1-2). 

The study team recruited 48 experts in content and/ or methodology, screened them for conflicts of 
interest, and trained them to use the rubrics for evaluating the quality and relevance of IES- 
published FRP reports and corresponding proposals. Training of panelists occurred in January and 
February 2010. Comments from panelists during the training were used to make minor revisions to 
each rubric. A total of 79 panels were organized to review FRPs, including 46 experts and an average 
of 2.92 experts per panel. The number of panels on which any one expert served averaged 5.02 and 
ranged from 2 to 10. Three panels reviewed a report with no corresponding proposal, since no 
written initial proposal for the FRP was available. All other panels reviewed both the proposal for 
the FRP and the resulting IES-published report. Five panels reviewed multiple IES-published 
reports that arose from the same proposal. While three experts were generally assigned to review 
each set of FRP documents based on content or methodology expertise, in five cases, fewer experts 
were assigned to review a product because no other panelists had the necessary expertise. Ratings 
for each FRP document (proposal or report) were generated by averaging the scores across all 
panelists and across dimension-specific indicators. Panel reviews and reconciliation phone calls 
occurred between February and May 2010. 

On average, the expert panels rated IES-published FRP reports and corresponding proposals as 
being between “adequate” and “strong” in quality. The IES-published reports received a mean 
quality rating of 3.81 on a 5-point scale, while the corresponding FRP proposals received a mean 
quality rating of 3.24. For proposals, 9.1 percent of quality ratings were at the highest level of quality 
(“very strong”), while for IES-published FRP reports, 26.9 percent of quality ratings were “very 
strong” (Figure 1-1). 
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Figure 1-1. Distribution of expert panelist indicator-level quality and relevance ratings for IES- 
published Fast Response Project reports and corresponding proposals from all 10 
RELs combined 
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Figure Reads: 9.1 percent of the indicator-level proposal quality ratings submitted by expert panelists had a value of 5 (“very strong") on 
a 5-point scale. 

NOTE: The distributions for proposal quality and relevance were based on 1,280 and 656 indicator-level ratings, respectively, which is 
largely attributed to differences in number of indicators associated with quality and relevance. The distributions for lES-published 
report quality and relevance were based on 2,051 and 1,065 indicator-level ratings, respectively. The difference in number of 
indicators associated with quality and relevance between lES-published reports and proposals is due to the fact the number of lES- 
published reports is larger than that of proposals. Ratings of “not applicable” were not included in these frequency distributions. 

Source: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports 
prepared under the 2006-2011 REL contracts and published by IES by September 1, 2011); Rubric and Scoring Form for the Quality 
and Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals for impact studies under the 2006-2011 
REL contracts submitted by September 1, 2011) (Appendix A). 


The expert panels rated IES-published FRP reports and corresponding proposals, on average, as 
being between “adequate” in relevance and “relevant.” The IES-published reports received a mean 
relevance rating of 3.64 on a 5-point scale, while the corresponding FRP proposals received a mean 
relevance rating of 3.39. For proposals, 13.0 percent of relevance ratings were at the highest level of 
relevance (“highly relevant”), while for IES-published reports, 19.3 percent of relevance ratings were 
at the level of “highly relevant” (Figure 1-1). The interim report provides more detailed REL-specific 
findings on the quality and relevance of the FRP reports and corresponding proposals. 


Report Organization 


This final report addresses the remaining two questions for the evaluation, relying on expert panel 
review of REL impact study reports published by September 1, 2011, and impact study proposals 
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submitted by that date, and on a survey of state educational agency and local school district 
administrators conducted between October 2011 and May 2012. It is organized as follows. Chapter 
2 describes the design of the evaluation and specific methods used to address the final two research 
questions. Data on the impact study proposals and reports published by IES, and expert panel 
reviews of the quality and relevance of those products, are presented in chapter 3 along with results 
from a survey of REL customers on the relevance and usefulness of REL products and services. 
Ten subsequent chapters describe the quality and relevance of each REL’s impact study reports and 
corresponding proposals as well as results of the customer survey based on each region’s 
respondents. Appendix A includes the rubric used by the expert panel, and Appendix B describes 
the process used for assessing inter-rater agreement among expert panel members. Appendix C 
provides the distribution of indicator-level expert panelist ratings for REL impact study proposals 
and IES-published reports on quality and relevance. Appendix D includes a summary of the REL 
impact study methods and results. Appendix E describes the sample and weights for the customer 
survey. Appendix F includes the customer survey administered to state and local education agency 
administrators, and Appendix G lists the Technical Working Group (TWG) members advising the 
evaluation study team. 
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This report addresses the following two research questions: 

■ What were the technical quality and relevance of REL impact study reports published 
by IES and of the corresponding proposals? 

■ How relevant and useful were REL technical assistance products to the needs of states 
and districts in their regions? 

The study team for the evaluation used a variety of data collection and analysis activities to address 
these questions. Plans for these activities were developed in consultation with members of a 
Technical Working Group with expertise in program evaluation and education-related technical 
assistance (Appendix G), and with technical guidance from the project officer and leadership within 
the Evaluation Division of the NCEERA. 

To evaluate the technical quality and relevance of impact study reports and proposals, the study 
team organized expert panels consisting of individuals with relevant content and/ or methodological 
expertise to rate the technical quality and relevance of impact study reports published by IES by 
September 1, 2011, and of the corresponding initial proposals. 


Expert Panel Review of Impact Studies 

Section 186(c) of ESRA requires all research, statistics, and evaluation reports conducted by, or 
supported through, IES, to “be subjected to rigorous peer review before being published or 
otherwise made available to the public.” In the case of REL impact study reports, this review 
occurred through an Analytical and Technical Support (ATS) contract, which NCEERA awarded to 
Mathematica Policy Research, Inc., on September 29, 2006 and ended on March 28, 2012. A REL 
would first submit a proposal for the impact study to its NCEERA project officer for review. 
Additional review of the proposal occurred through anonymous external reviewers working under 
the ATS contract. These reviews informed NCEERA’s decision to reject a proposal, approve the 
project, or request the REL to revise its proposal. For approved projects, RELs would perform the 
work and submit draft reports to the NCEERA project officer for review. The ATS would again 
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conduct an external review of the report. Also, at this point in the process the NCEERA 
Commissioner would transmit the report to the IES Standards and Review Office (SRO) for 
external review, resulting in revisions, and ultimately, the decision by the SRO office whether to 
approve the report for publication by IES. 

RELs submitted initial as well as revised proposals to the IES Knowledge Utilization Division as 
part of the impact study process. The evaluation study team chose to focus the proposal review on 
initial proposals. While later proposals could be seen as representing the collective work of the REL, 
IES, and its ATS contractor, the initial proposals could be more completely attributed to the RELs. 
In addition, by evaluating the initial proposals and IES-published reports, the study is able to 
document ratings before and after the contributions of the IES review process. Because of resource 
limitations for the evaluation, the evaluation study team and the NCEERA decided to focus panel 
reviews on the beginning and end of IES-published impact study reports. Revised proposals, 
proposals that were rejected or were still under review, and proposals for reports that were not 
published by IES were all excluded from panel review. Consequently, findings on the quality and 
relevance of proposals refer only to proposals resulting in IES-published impact study reports that 
were published between March 1, 2006 and September 1, 2011. 

The evaluation study team collected from REL web sites and from the REL program office all of 
the impact study reports published by IES on its website, as well as the corresponding initial 
proposals that were produced by the 10 RELs between March 1, 2006 to September 1, 201 1. 12 A 
total of eight impact studies had been completed; they resulted in eight impact study reports 
published by seven RELs. 1 ’ To ensure that all 10 RELs had impact-related documents for review, 
the study team also collected one initial proposal for each of the three RELs that did not have a 
published report but whose final report was close to publication. This resulted in 8 reports and 1 1 
initial proposals for the expert panel review (Table 2-1). All of the studies reviewed used randomized 
controlled trials (RCTs). 


12 It is important to note that the REL program contract start dates varied by REL as follows: REL Appalachia, February 
6, 2006; REL Central, January 20, 2006; REL Mid-Atlantic, March 23, 2006; REL Midwest, March 9, 2006; REL 
Northeast, March 15, 2006; REL Northwest, February 1, 2006; REL Pacific, March 16, 2006; REL Southeast, March 
16, 2006; REL Southwest, March 15, 2006; and REL West, January 18, 2006. IES extended the contract end dates 
through December 31, 2011. 

13 Under the 2006-2011 REL contracts, the RELs completed a total of 23 impact studies, which resulted in 24 impact 
study reports (two reports were published about one study). This included eight reports about the eight completed 
impact studies that were published before September 1 , 201 1 and were part of the review conducted for this 
evaluation. There were 15 impact studies completed and 16 reports published after September 1, 2011 that were not 
part of the review for this evaluation. 


Evaluation of the Regional Educational Laboratories: Final Report 

14 



Evaluation Design and Methods 


2 


Table 2-1 . Impact study proposals and reports by REL 


REL 

Study title 

AP 

Effects of the Kentucky Virtual Schools' Hybrid Program for Algebra 1 on Grade 9 Student Math 
Achievement * 

CE 

Classroom Assessment for Student Learning: Impact on Elementary School Mathematics in the 
Central Region: Final Report 

MA 

A Multisite Cluster Randomized Trial of the Effects of Compass Learning Odyssey Math on the 
Math Achievement of Selected Grade 4 Students in the Mid-Atlantic Region 

MW 

Improving Adolescent Literacy Across the Curriculum in High Schools (Content Literacy Continuum, 
CLC)* 

NE 

Impact of the Thinking Reader Software Program on Grade 6 Reading Vocabulary, 
Comprehension, Strategies, and Motivation: Final Report 

NW 

An Experimental Study of the Project CRISS Reading Program on Grade 9 Reading Achievement in 
Rural High Schools 

PA 

Pacific Evaluation of Principles-Based Professional Development to Improve Reading 
Comprehension for English Language Learners * 

SE 

Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten 

SW 

The Impact of Collaborative Strategic Reading on the Reading Comprehension of Grade 5 
Students in Linguistically Diverse Schools 

WE 

Effects of Problem Based Economics on High School Economics Instruction 

WE 

Accommodations for English Language Learner Students: The Effect of Linguistic Modification of 
Math Test Item Sets 


*Proposal only. 


The study team developed two rubrics to assist in the expert panel review of impact studies: one for 
IES-published reports and one for initial proposals (Appendix A). The rubrics included two 
dimensions: technical quality and relevance. Each dimension was further defined by multiple 
indicators (Table 2-2). Indicators for the dimensions were rated on a 5-point scale, with 1 being the 
lowest rating and 5 being the highest rating. Descriptive “anchors” were provided for the extreme 
and median points on the scale (values 1, 3, and 5). While the indicators and anchors were designed 
to enhance the reliability of the ratings, expert panel members were also required to use their 
professional judgment when assigning ratings. For example, for impact study report indicator ID, 
reviewers were not provided guidance about what would be considered a “rigorous” random 
assignment design and implementation. The panel members’ professional experience and training as 
What Works Clearinghouse reviewers was considered critical to ensuring their common 
understanding of research standards and terminology. 

The draft rubrics, including the anchors, underwent a series of external reviews and tests. First, the 
draft mbrics were pilot tested by two reviewers. Cognitive interviews of pilot-test reviewers were 
used to assess their understanding of the mbrics and rationale for the ratings. The rubrics were 
further refined by incorporating feedback from the expert panel after the training sessions described 
in the next section. 
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Table 2-2. Dimensions and indicators from the rubric used in the expert panel review of IES- 
published impact study reports and corresponding proposals 


Proposals: Dimensions and Indicators 

Dimension: Quality 

Indicator 1A. 

The intervention for the treatment group and the condition for the control group are clearly 
described. 

Indicator IB. 

The plan to document implementation of the intervention (e.g., exposure, quality of delivery, 
adherence) is appropriate. 

Indicator 1C. 

The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

Indicator ID. 

The design for random assignment is rigorous. 

Indicator IE. 

The sampling strategies are appropriate for targeted populations, and the resulting sample 
size(s) for the impact questions will have adequate statistical power 

Indicator IF. 

Outcome measures are valid, reliable, and not overly-aligned with the intervention. 

Indicator 1G. 

The data collection plan is appropriate for the research questions. 

Indicator 1H. 

The data analyses will use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

Indicator 11. 

The proposed research is feasible to be carried out with reasonable timeline and sequencing 
of activities. 

Dimension: Relevance 

Indicator 2A. 

The proposal provides a strong justification why the topic was selected as salient to meet 
the needs of the region served by the REL. 

Indicator 2B. 

The proposal provides a thorough summary of key literature and/or previous research in the 
topic area. 

Indicator 2C. 

The proposal provides a strong justification for selecting the particular intervention that is 
being studied. 

Indicator 2D. 

The report will contribute new information about the effectiveness of the intervention being 
studied and the more general topic being addressed. 

Indicator 2E. 

The proposal is clearly-written and well-presented. 

Reports: Dimensions and Indicators 

Dimension: Quality 

Indicator 1A. 

The intervention for the treatment group and the condition for the control group are clearly 
described. 

Indicator IB. 

Implementation of the intervention is well documented (e.g., exposure, quality of delivery, 
adherence). 

Indicator 1C. 

There is minimal contamination in the form of crossover between subjects in treatment and 
control condition or spillover of the intervention from the treatment to the control group. 

Indicator ID. 

The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

Indicator IE. 

The design and implementation of random assignment are rigorous. 

Indicator IF. 

The sampling strategies are appropriate for targeted populations and the resulting sample 
size(s) for the impact questions have adequate statistical power. 

Indicator 1G. 

Outcome measures are valid and reliable and not overly-aligned with the intervention. 

Indicator 1H. 

The data collection plan is appropriate for the research questions. 

Indicator 11. 

The data collection plan is well-implemented. 

Indicator 1J. 

The overall attrition rate and differential attrition rates are acceptable given the length of the 
intervention. 
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Table 2-2. Dimensions and indicators from the rubric used in the expert panel review of IES- 
published impact study reports and corresponding proposals (continued) 


Reports: Dimensions and Indicators (continued) 

Dimension: Quality (continued) 

Indicator IK. 

The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

Indicator 1L. 

Appropriate statistics are provided to describe the sample and support the findings. 

Indicator 1M. 

The conclusions about the intervention are drawn appropriately and consistently. 

Indicator IN. 

All of the research questions are specifically addressed by the analyses. 

Indicator 10. 

The limitations of the study are clearly and comprehensively stated. 

Dimension: Relevance 

Indicator 2A. 

The report provides a strong justification why the topic was selected as salient to meet the 
needs of the region served by the REL. 

Indicator 2B. 

The report provides a thorough summary of key literature and/or previous research in the 
topic area. 

Indicator 2C. 

The report provides a strong justification for selecting the particular intervention that is 
being studied. 

Indicator 2D. 

The report contributes new information about the effectiveness of the intervention being 
studied and the more general topic being addressed. 

Indicator 2E. 

The executive summary of the report is easy to read and understand for a lay audience. 

Indicator 2F. 

The report is clear and well-written for the technical audience. 


An indicator- specific comment field was provided to record raters’ notes, and a general comment 
field permitted reviewers to indicate uncertainty about the rating or note strengths or weaknesses 
not linked to a specific indicator. 


To identify potential expert panel members, the evaluation team began with a list of What Works 
Clearinghouse certified reviewers provided by IES. To avoid conflicts of interest, the evaluation 
team excluded from further consideration potential reviewers who had worked on the 
corresponding REL studies under review. The team subsequently reviewed the bios of potential 
panelists to identify areas of content expertise and prioritized for recruitment those with expertise 
that corresponded with the topics covered in the REL impact studies. 


The evaluation study team recmited 1 2 experts in content and methodology, screened them for 
conflicts of interest, and trained them to use the rubrics for evaluating the quality and relevance of 
IES-published impact study reports and corresponding proposals. Panelists were primarily 
academics but also included contractors with expertise relevant to the content of the impact studies 
and/ or methods used. All 12 experts participated in one of two 1-day, in-person training sessions in 
Rockville, MD in November 2011. The specific purposes of the training were to inform expert 
panelists about their roles and responsibilities; teach them about the review process; help them 
become familiar with the scoring mbrics; and provide opportunities to practice scoring products 
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using the rubrics, debrief, and undertake reconciliation with other experts. Comments from panelists 
during the training were used to make minor revisions to each rubric. 

A total of 19 panels were organized to review impact studies (8) and proposals (11), including 12 
experts and three experts per panel. The number of panels on which any one expert served averaged 
5.2 and ranged from 2 to 7. All panels reviewed both the proposal and the resulting IES-published 
report, except for the three proposals with no corresponding report. All the expert panelists 
submitted their ratings for each of the IES-published reports and corresponding proposals assigned 
to them. Ratings for each document (proposal or report) were generated by averaging the scores 
across all panelists and across dimension- specific indicators. 

To assess the reliability of the expert panel reviews, the study team measured inter-rater agreement 
by product type (i.e., proposals and reports) and by dimension (i.e., quality and relevance) using the 
“Rwg®” index (LeBreton and Senter 2008). Rwg® values range from 0 to 1 and provide estimates 
of the level of inter-rater agreement between sets of experts who provide ratings for different targets 
(i.e., IES-published reports and proposals in this case) using multiple items. Levels of Rwg® can be 
interpreted as follows (LeBreton and Senter 2008): 0.00-0.30 (lack of agreement), 0.31-0.50 (weak 
agreement), 0.51-0.70 (moderate agreement), 0.71-0.90 (strong agreement), 0.91-1 (very strong 
agreement). 14 

While, in general, the initial inter-rater agreement was high (e.g., a mean Rwg® higher than 0.85), the 
study team held one reconciliation discussion with a panel whose review of a proposal had an Rwg® 
of less than 0.30. In the reconciliation session, all three members convened over a conference call 
that was facilitated by a study team researcher. Prior to the call, experts received email notification, 
including their original ratings. Panelists did not see the ratings of the other members of their panel, 
since confidentiality of ratings was assured to panelists. Only the dimensions and indicators with low 
agreement were reconciled. On the call, experts were asked to discuss the strengths and weaknesses 
of the proposal relative to the indicator in question. Experts were given an opportunity to revise 
their scores voluntarily after the reconciliation call. Experts participating in reconciliation used the 
on-line rating system to modify their scores or informed the study team within 3 business days of 
the reconciliation call that no changes were being made. After reconciliation, mean inter-rater 
agreement was uniformly high (Table 2-3). Further analysis of the Rwg® ratings indicated that they 
were negatively skewed for proposals and that the median Rwg® for proposals was higher than the 
mean (e.g., “Quality” ratings on proposals had a median Rwg® of 0.93 vs. a mean of 0.90). 


14 Additional information on the use of the Rwg© can be found in Appendix B. 
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Table 2-3. Rwg(j) statistics for reviews of lES-published impact studies and corresponding 
proposals 


Product 

Dimension 

Quality 

Relevance 

Proposals (ZV=11) 

0.90 

0.89 

Reports (N= 8) 

0.96 

0.95 


Table Reads: For the 11 proposals, the mean inter-rater agreement for indicator ratings along the quality dimension was 0.90. 


Source: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Studies (For Reports); 
Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Studies (For Proposals) 
(Appendix A). 


Limitations of Expert Panel Review 

The expert panel review of impact study reports and proposals was characterized by several 
limitations in terms of the generalizability of the findings. Impact study proposals that did not result 
in IES publications by September 1 , 201 1 and draft impact study reports or reports published after 
September 1, 201 1 were not included in this review. Therefore, results of the expert panel review 
cannot be generalized to reports that were not included in the review. 

Readers are cautioned that it is not possible to use the expert panel reviews of IES-published impact 
study reports and initial proposals to distinguish the contributions of the REL, the ATS contractor, 
or NCEERA to the quality or relevance ratings for the reports published by IES. Readers should 
also be aware that, although there is correspondence between the 8 IES-published impact study 
reports reviewed and 8 of the 1 1 initial proposals, the average indicator-level quality and relevance 
ratings are distributed differently for reports than for proposals, for two reasons. First, because three 
initial proposals were without a corresponding report, the average ratings are distributed differently 
across impact studies for reports and for proposals. Second, because the rubric for reviewing reports 
had more indicators of quality and relevance than the mbric for rating proposals, the average ratings 
are distributed differently across indicators for reports and for proposals. 


REL Customer Survey 

To evaluate the relevance and usefulness of REL research and/or technical assistance products to 

the needs of the state and district administrators nationwide and within their regions, in 

winter/ spring 2012, the study team conducted a survey of a nationally representative sample of state 
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and district administrators. 15 The survey addressed respondents’ needs for education research 
and/or technical assistance, familiarity with and use of REL products and services, and satisfaction 
with the RELs. The statutory mission of the RELs includes both research and technical assistance, 
so the survey included both concepts together to encompass the broad range of activities and needs 
addressed by the RELs. 1 


Sample Design for Survey of REL Customers 

The primary customers of the RELs are employees of SEAs and LEAs in their respective 
geographic regions. However, the study team included in the survey population not only actual users 
of REL services and products but also potential users. Surveying potential users instead of only actual 
users allowed us to glean information about the target audiences’ needs, awareness of REL services 
and products, and the reasons for non-use. 

All SEAs and defined users within the SEAs were included in the sample frame. For LEAs, a two- 
stage sample design was used, where a sample of LEAs was first selected and then users were 
sampled from lists compiled from district websites. A unique feature of REL Pacific is that all SEAs 
served by the REL have only one school district, so the SEAs and LEAs coincide. Therefore, there 
is no LEA stratum for the Pacific region. To build the list of targeted respondents for SEAs in 10 
regions and the sample frame for LEAs in 9 regions (excluding Pacific), the study team used state 
and district websites to identify administrators holding the following state and district positions: 

■ Chief State School Officers/Superintendents 

■ Deputy Superintendents 


15 Based on input from a Technical Working Group (TWG), the evaluation team identified state and district 
administrators as the primary audience or customers for REL products and services. 

16 The survey, which can be found in Appendix F of the report, did not define the terms research and technical 
assistance. However, Section 102 of ESRA defines applied research as “research — (A) to gain knowledge or 
understanding necessary for determining the means by which a recognized and specific need may be met; and (B) that 
is specifically directed to the advancement of practice in the field of education.” It defines basic research as 
“research — (A) to gain fundamental knowledge or understanding of phenomena and observable facts, without 
specific application toward processes or products; and (B) for the advancement of knowledge in the field of 
education.” It defines technical assistance as “(A) assistance in identifying, selecting, or designing solutions based on 
research, including professional development and high-quality training to implement solutions leading to — (i) 
improved educational and other practices and classroom instruction based on scientifically valid research; and (ii) 
improved planning, design, and administration of programs; (B) assistance in interpreting, analyzing, and utilizing 
statistics and evaluations; and (C) other assistance necessary to encourage the improvement of teaching and learning 
through the applications of techniques supported by scientifically valid research.” 


Evaluation of the Regional Educational Laboratories: Final Report 

20 



Evaluation Design and Methods 


2 


■ Directors of 

— Special Education, 

Title I, 

— Bilingual Education/ESL, 

— Curriculum and Instruction, 

— Certification, 

— Professional Development, 

— Assessment, 

— School Improvement, 

— Migrant Education, 

— Gifted Education, 

— Early Childhood Services, 

— Career and Technology Education. 

This approach did not cover all conceivable actual and potential users but rather the majority of 
them; it would be difficult to identify all users, and the cost would have been prohibitive. Likewise, 
while principals and teachers may be perceived as potential users of REL products and services, 
inclusion of school level personnel was considered prohibitively expensive. Furthermore, the 
proportion of school-level personnel who were actual users instead of potential users was expected 
to be small, so a very large sample would have been required to elicit a sufficient number of 
responses for analysis of survey items pertaining to services received. 

The universe was naturally stratified by SEA and LEA, but stratification was also necessary to use 
different sample designs for the SEAs and LEAs. This type of stratification was also viewed as 
beneficial because user needs may differ for state and local personnel. Furthermore, the universe was 
stratified by REL to produce estimates by REL with a specified level of precision . 1 


17 The target precision was set to be 3 percentage points for a population proportion of 50 percent for a user 
characteristic. 
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Data Collection for Survey of REL Customers 

Data collection for the web-based customer survey began in fall 2011. To gather information on 
response rates and determine if any necessary adjustments would need to be made to the design or 
to the projected precision, data were initially collected on the state sample and a sub-sample of the 
district administrators. In October 2011, the study team sent emails to all 346 state administrators 
and a sub-sample of 1,027 district administrators. In February 2012, emails were sent to state and 
district administrators who had not responded to the survey in fall 201 1 /winter 2012 and to the 
remaining 3,800 district administrators in the sample who were not included in the sub-sample. The 
emails included: 

■ A link to the customer survey, 

■ A username and password for accessing the survey, 

■ A letter from ED requesting the administrator’s participation in the survey and 
explaining the purpose of the survey, and 

■ A letter from Westat providing instructions for completing the survey and contact 
information for the study team. 

Administrators were asked to complete the survey within 2 weeks. Those who did not respond after 
2 weeks were sent a reminder email and asked to respond within a week. Once that date had passed, 
those who had not responded were given to Westat’s Telephone Research Center (TRC) for 
telephone follow-up. The TRC called to remind administrators to complete the survey online and 
also offered to complete the survey with the administrators over the phone. A third of all the 
completed surveys were conducted over the phone, with the phone interviewer entering the 
administrators’ responses into the web survey. 1S Through online searches and calls to the state or 
district departments of education, the study team traced administrators as needed to obtain updated 
contact information. The TRC continued to contact administrators until data collection ended in 
early May 2012. The field results of the LEA sample and the SEA sample are shown in Table 2-4. 


18 Regression analyses demonstrated that after controlling for respondents’ regions, the results for two key items in the 
survey, item B1 (“How familiar are you with the REL program overall?”) and item B8 (“Overall, how satisfied were 
you with the work of the REL program nationwide including your regional REL?” ) did not vary significantly by 
differences in mode of survey completion (online versus telephone) (state administrators for B1 \p — 0.558; state 
administrators for B8 : p =0.595; district administrators for B \\p — 0.872; district administrators for B8 : p — 0.331) . 
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Table 2-4. Results for state and district administrators on the REL customer survey, by 
response status 


Status code 

State 

administrator 

sample 

% of potentially 
eligible SEA 
administrators 

District 

administrator 

sample 

% of potentially 
eligible LEA 
administrators 

Complete 

290 

87 

3,709 

80 

Nonresponse 

12 

4 

196 

4 

Eligibility unknown 

30 

9 

736 

16 

Total potentially eligible 

332 

100 

4,641 

100 

Ineligible 

14 

4 

181 

4 


Note: Administrators were deemed ineligible if they were no longer in the position and the position was vacant, were inadvertently 
duplicated, the person sampled was not actually an administrator or was an inappropriate respondent to the survey (e.g., an 
administrative assistant or Director of Facilities). 


There were 766 users (14.8 percent of the sample) who did not respond at all by the close of data 
collection, even after repeated email reminders and phone calls, and their eligibility was not 
ascertained. The overall response rate was 87 percent for the sample of state administrators, and 80 
percent for the sample of district administrators. The response rates by REL are shown in Table 2-5. 
Information on sampling and weighting for the customer survey are presented in Appendix E. 


Table 2-5. Survey sample sizes and response rates by REL 



SEA 

LEA 

REL Name 

Sample size 

Response rate (%) 

Sample size 

Response rate (%) 

Appalachia 

28 

100 

534 

83 

Central 

44 

91 

511 

82 

Mid-Atlantic 

32 

90 

543 

78 

Midwest 

47 

91 

546 

81 

Northeast and Islands 

46 

80 

538 

73 

Northwest 

33 

91 

474 

82 

Pacific 

17 

71 

NA 

NA 

Southeast 

40 

95 

596 

81 

Southwest 

33 

72 

566 

81 

West 

26 

83 

526 

78 

Total 

346 

87 

4,834 

80 
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Cross-REL Findings on the Quality, Relevance, 
and Usefulness of Research and Technical 

Assistance Products 



This chapter focuses on the quality, relevance, and usefulness of REL research and technical 
assistance products by addressing the following two research questions: (1) What were the technical 
quality and relevance of REL impact study reports published by IES and of the corresponding 
proposals? and (2) How relevant and useful were the REL technical assistance products to the needs 
of states and districts in their regions? 


What Were the Technical Quality and Relevance of REL Impact 
Study Reports Published by IES and of the Corresponding 
Proposals? 

To assess the technical quality and relevance of REL impact study reports and proposals, the study 
team organized expert panel reviews of 1 1 impact study proposals submitted by the 10 RELs 
between March 1, 2006, and September 1, 2011, and 8 impact study reports from 7 RELs published 
by IES between November 2009 and April 2011. Three RELs had not produced any IES-published 
impact study reports by September 1 , 201 1. 

The expert panelists rated eight IES-published impact study reports as, on average, 
between “strong” and “very strong” in quality (mean quality rating of 4.10 on a 5-point 
scale, with 5 being the highest value). The 11 impact study proposals selected for review, 
including proposals for the 8 studies producing IES-published reports, received a mean 
quality rating of 3.59, which is between “adequate” and “strong” on a 5-point scale. For 
reports, all indicators had a rating of 4.0 or higher in quality except “The intervention for the 
treatment group and the condition for the control group are clearly described,” “Implementation of 
the intervention is well documented (e.g., exposure, quality of delivery, adherence),” “The design 
and implementation of random assignment are rigorous,” “The sampling strategies are appropriate 
for targeted populations, and the resulting sample size(s) for the impact questions have adequate 
statistical power,” and “The overall attrition rate and differential attrition rates are acceptable given 
the length of the intervention.” Indicator-specific ratings of quality are provided in Table 3-1. 
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The expert panelists rated eight IES-published reports as, on average, between “relevant” 
and “very relevant” (mean relevance rating of 4.06 on a 5-point scale, with 5 being the 
highest value). The 11 impact study proposals selected for review received a mean relevance 
rating of 3.61, which is between “adequate” in relevance and “relevant.” For reports, all 
indicators had a rating of 4.0 or higher in relevance except “The report provides a strong 
justification for selecting the particular intervention that is being studied,” and “The executive 
summary of the report is easy to read and understand for a lay audience.” Ratings of IES-published 
impact study reports and proposals by specific indicators of relevance are provided in Table 3-1. The 
separate chapters for each REL discuss detailed REL-specific expert panel review findings. 

Table 3-1. Mean ratings (on a 5-point scale, with 5 being the highest) from expert panel review 
of REL impact study proposals and IES-published reports, by rating indicator 


Quality indicators for proposals 

Proposals 
(/V= 11) 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

3.61 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

3.82 

ID. The design for random assignment is rigorous. 

3.64 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.55 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.15 

1G. The data collection plan is appropriate for the research questions. 

3.76 

1H. The data analyses will use appropriate methods to address the research questions 
(e.g., adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

3.45 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

4.00 

Relevance indicators for proposals 

Proposals 
(N= 11) 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

3.70 

2B. The proposal provides a thorough summary of key literature and/or previous research in 
the topic area. 

3.27 

2C. The proposal provides a strong justification for selecting the particular intervention that 
is being studied. 

3.33 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.06 

2E. The proposal is clearly written and well presented. 

3.67 
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Table 3-1. Mean ratings (on a 5-point scale, with 5 being the highest) from expert panel review 
of REL impact study proposals and lES-published reports, by rating indicator 
(continued) 



Quality indicators for reports 

lES-published 

reports 

(/V=8) 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.96 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

3.83 

1C. There is minimal contamination in the form of crossover between subjects in treatment 
and control condition or spillover of the intervention from the treatment to the control 
group. 

4.33 

ID. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

4.50 

IE. The design and implementation of random assignment are rigorous. 

3.83 

IF. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s)for the impact questions have adequate statistical power. 

3.79 

1G. Outcome measures are valid and reliable and not overly aligned with the intervention. 

4.00 

1H. The data collection plan is appropriate for the research questions. 

4.08 

11. The data collection plan is well implemented. 

4.25 

U. The overall attrition rate and differential attrition rates are acceptable given the length 
of the intervention. 

3.83 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

4.21 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.33 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

4.38 

IN. All of the research questions are specifically addressed by the analyses. 

4.21 

10. The limitations of the study are clearly and comprehensively stated. 

4.00 

Relevance indicators for reports 

lES-published 

reports 

(/V=8) 

2A. The report provides a strong justification why the topic was selected as salient to meet 
the needs of the region served by the REL. 

4.33 

2B. The report provides a thorough summary of key literature and/or previous research in 
the topic area. 

4.00 

2C. The report provides a strong justification for selecting the particular intervention that is 
being studied. 

3.75 

2D. The report contributes new information about the effectiveness of the intervention being 
studied and the more general topic being addressed. 

4.17 

2E. The executive summary of the report is easy to read and understand for a lay audience. 

3.88 

2F. The report is clear and well-written for the technical audience. 

4.25 


NOTE: Proposals were submitted by the RELs between March 1, 2006, and September 1, 2011; reports were published by IES between 
November 2009 and April 2011. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports 
prepared under the 2006-2011 REL contracts and published by IES by September 1, 2011); Rubric and Scoring Form for the Quality 
and Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals for impact studies under the 2006-2011 
REL contracts submitted by September 1, 2011) (Appendix A). 


Evaluation of the Regional Educational Laboratories: Final Report 

27 



Cross-REL Findings on the Quality, Relevance, 
and Usefulness of Research and Technical 
Assistance Products 


3 


A summary of the methods and results for each of the 8 IES-published impact studies is provided in 
Appendix D. 19 Each summary contains (1) study citation, (2) what the study was about, (3) features 
of the intervention, and (4) what the study found. 


How Relevant and Useful Were the REL Technical Assistance 
Products to the Needs of States and Districts in Their Regions? 

Between the fall of 201 1 and spring of 2012, the evaluation team surveyed state and district 
administrators to ask them how relevant and useful REL technical assistance products were in 
meeting their needs 2 ". State and district administrators were included in the sample regardless of 
previous use of REL services or familiarity with the REL program. Specific research questions 
included: 


■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators was teacher/staff evaluation (53 percent). The most 
commonly reported area of “high need” for education research and/or technical assistance 
among district administrators was content standards, curriculum, or instruction in science, 
technology, engineering, and mathematics (STEM) (37 percent) (Table 3-2). 


19 The summaries for impact studies were provided by IES. 

20 Appendix F includes the survey administered to state and local education agency administrators. 
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■ Administrators were asked to indicate whether they had a “high need,” “moderate 
need,” or “low or no need” for research and/ or technical assistance in specific topic 
areas. The areas in which the five largest percentages of state administrators indicated a 
“high need” were teacher/ staff evaluation (53 percent), achievement gaps (50 percent), 
college or career readiness (49 percent), support for low-achieving schools (49 percent), 
and using data for decisions (47 percent). (Table 3-2 for top five needs and Table 3-3 
for a complete set of needs.) 

■ The areas in which the five largest percentages of district administrators indicated a 
“high need” for research and/ or technical assistance were content standards, 
curriculum, or instruction in STEM (37 percent); using data for decisions (35 percent); 
achievement gaps (34 percent); content standards, curriculum or instruction in 
reading/writing (32 percent); and assessment (31 percent). (Table 3-2 for top five needs 
and Table 3-3 for a complete set of needs.) 

Table 3-2. Percentage of state and district administrators who reported various levels of need 
for different types of research and/or technical assistance: Top five needs— National: 
School year 2011-12 


Type of research and/or technical 
assistance 

n 

High need for 
research and/or 
assistance 

Moderate need 
for research 
and/or 
assistance 

Low or no need 
for research 
and/or 
assistance 

State administrators 

Teacher/staff evaluation 

286 

53 

31 

16 

Achievement gaps 

288 

50 

37 

13 

Support for low-achieving schools 

286 

49 

37 

15 

College or career readiness 

286 

49 

37 

14 

Using data for decisions 

286 

47 

40 

14 


District administrators 

Content standards, curriculum or 
instruction in: STEM 

3,692 

37 

45 

18 

Achievement gaps 

3,698 

35 

47 

19 

Using data for decisions 

3,689 

35 

44 

21 

Content standards, curriculum or 
instruction in: reading/writing 

3,686 

32 

48 

21 

Assessment (formative or summative) 

3,695 

31 

47 

22 


NOTE: Excludes Other. See Table 3-3 for a complete list of topic areas in which respondents reported needs for research and/or technical 
assistance. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


Twenty-nine percent of state administrators and 26 percent of district administrators 
reported that their research and technical assistance needs were met “very well” (as 
opposed to “moderately well” or “not well”), regardless of the source of assistance. 
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Table 3-3. 


Percentage of state and district administrators who reported various levels of need for different types of research and/or 
technical assistance— National: School year 2011-12 


Type of research and/or technical 
assistance 

State administrators 

District administrators 


Need for research and/or assistance 


Need for research and/or assistance 

High 

Moderate 

Low or no need 

High 

Moderate 

Low or no need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

288 

50 

37 

13 

3,698 

34 

47 

19 

Assessment (formative or summative) 

286 

37 

45 

19 

3,695 

31 

47 

22 

Behavior, character education, or health 

284 

12 

37 

51 

3,681 

18 

44 

38 

College or career readiness 

286 

49 

37 

14 

3,677 

30 

43 

27 

Content standards, curriculum or 
instruction in STEM 

285 

37 

41 

22 

3,692 

37 

45 

18 

Content standards, curriculum or 
instruction in reading/writing 

284 

33 

40 

26 

3,686 

32 

48 

21 

Content standards, curriculum or 
instruction in other subject 

284 

13 

48 

39 

3,666 

11 

51 

38 

Dropout prevention 

284 

40 

42 

18 

3,684 

24 

37 

39 

Early childhood 

284 

33 

39 

28 

3,687 

19 

39 

41 

English language learners 

284 

44 

41 

15 

3,688 

23 

38 

39 

High school reform 

283 

37 

38 

25 

3,687 

23 

38 

39 

Leadership 

285 

37 

42 

21 

3,688 

22 

51 

27 

Longitudinal data systems 

283 

35 

36 

29 

3,691 

28 

45 

27 

Parental involvement 

282 

31 

45 

23 

3,690 

26 

47 

27 

Professional development 

287 

38 

44 

17 

3,693 

27 

51 

23 

Rural schools 

285 

35 

39 

26 

3,685 

20 

28 

52 

School accountability 

285 

38 

37 

25 

3,689 

17 

48 

35 

School choice 

284 

11 

33 

56 

3,683 

6 

25 

69 

School finance 

283 

17 

34 

49 

3,681 

18 

37 

45 

Students with disabilities 

283 

28 

44 

28 

3,687 

25 

49 

27 

Supplemental education services 

284 

17 

34 

48 

3,689 

12 

43 

45 

Support for low-achieving schools 

286 

49 

36 

15 

3,692 

26 

36 

38 

Teacher/staff evaluation 

286 

53 

31 

16 

3,694 

28 

45 

27 

Using data for decisions 

286 

47 

40 

14 

3,689 

35 

44 

21 

Other 

17 

* 

t 

88 

383 

33 

13 

54 


NOTES: Shaded cells are those that are mentioned in the text, t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 -spring 
2012 . 
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Figure 3-1. Percentage of all administrators who reported that their education research and 
technical assistance needs were met “very well," regardless of the source of 
assistance, by region: School year 2011-12 




NOTE: The total N for state administrators was 285, and the total N for district administrators was 3,686. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ The percentage of state administrators who reported that their needs were met “very 

well,” regardless of the source of assistance, varied by region, from a high of 60 percent 
in the Pacific to a low of 1 8 percent in the Midwest. The percentage of district 
administrators who reported that their needs were met “very well,” regardless of the 
source of assistance, also varied by region, from a high of 30 percent in the Southeast to 
a low of 18 percent in the Northwest (Figure 3-1). 


What Sources of Education Research and technical assistance do state and 
district administrators use? 

Forty-nine percent of state administrators and 18 percent of district administrators reported 
that they relied on the REL program “to a great extent” or “to a moderate extent” for 
education research and/ or technical assistance as opposed to a “small extent” or not at all. 
In contrast, 87 percent of state administrators relied on professional associations, and 82 
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percent of district administrators, relied on their counterparts in other districts “to a great 
extent” or “to a moderate extent.” 

Figure 3-2. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— National: School year 2011-12 


Sources of education research 
and technical assistance 

REL program nationwide - 

Comprehensive centers - 

Other federally funded providers - 

Other products or resources from ED,... - 

-T 

Professional associations - 
Colleges and universities - 

Consulting firms/contractors - 

-t 

Counterparts at other LEAs or SEAs - 
TA center supported by your state - 
Journals and publications - 
Other sources - 


JTS“ 


H 49 




H 62 


1ST 


TTT 


a 59 

□ 60 


70 


□ 59 
54 


T5 


1 45 


□ State 

□ District 


II 87 


178 


82 


J23_ 


1 59 


69 


74 


20 


40 


60 


80 


100 % 


NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works." The total Ns for state and 
district administrators on the item about reliance on the REL program were 289 and 3,700, respectively. The total N for state 
administrators on the items about other specified sources of research ranged from 284 to 288, depending on the number of state 
respondents who chose not to respond to an individual item. The total N for district administrators on the items about other specified 
sources of research ranged from 3,684 to 3,694, depending on the number of district respondents who chose not to respond to an 
individual item. The total N for district administrators for “Other sources" was 217. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ State and district administrators reported that they used a variety of sources for meeting 
their research and/ or technical assistance needs. Professional associations and 
counterparts in other states and districts were most commonly reported as being relied 
upon “to a great extent” or “to a moderate extent” (states: 87 and 78 percent, 
respectively; districts: 70 and 82 percent, respectively) (Figure 3-2). 

■ State administrators’ reliance on the REL program varied across the regions: the 
percentage who reported relying on the REL program for research and/ or technical 
assistance “to a great extent” or “to a moderate extent” ranged from a high of 71 
percent in the West to a low of 27 percent in the Southwest. For district administrators, 
the percentage who reported relying on the REL program “to a great extent” or “to a 
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moderate extent” ranged from a high of 22 percent in the Northwest to a low of 15 
percent in the Midwest (Figure 3-3). 

Figure 3-3. Percentage of all administrators who reported that they relied on the national REL 
program “to a great extent” or “to a moderate extent,” by region: School year 2011- 
12 




NOTE: The total N for state administrators was 289, and the total N for district administrators was 3,700. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


How easily do state and district administrators access the education research 
and/or technical assistance they need? 

Eighty-eight percent of state administrators and 93 percent of district administrators 
reported that it was “very easy” or “moderately easy” (as opposed to “not at all easy”) to 
access education research and/ or technical assistance across the available sources of 
information (Figure 3-4). 
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Figure 3-4. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— National: School year 2011-12 



Percent 

100 % 

80% 

60% 

40% 


20% 


0 % 


31 

Very easy 

39 





57 

Moderately easy 

54 


Not at all easy 

12 


8 


State 


District 


NOTE: The total N for state administrators was 288, and the total N for district administrators was 3,700. Percentages may not sum to 
100 due to rounding. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ The percentage of state administrators who reported that it was “very easy” to access 
education research and/ or technical assistance varied from 26 percent in the Midwest 
and West regions to 45 percent in the Mid-Atlantic region (Table 3-4). The percentage 
of district administrators who said it was “very easy” to access education research 
and/ or technical assistance varied from 30 percent in the Northwest region to 43 
percent in the Mid-Atlantic region. 
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Table 3-4. Percentage of all administrators who reported that it was “very easy” to access 

education research and/or technical assistance when needed, by region: School year 
2011-12 



Region 

n 

% of State administrators 

National 

93 

31 

Appalachia 

8 

30 

Central 

11 

29 

Mid-Atlantic 

12 

44 

Midwest 

11 

26 

Northeast and Islands 

14 

40 

Northwest 

9 

30 

Pacific 

t 

t 

Southeast 

15 

41 

Southwest 

7 

32 

West 

5 

26 

Region 

n 

% of District administrators 

National 

1,413 

39 

Appalachia 

160 

38 

Central 

161 

40 

Mid-Atlantic 

176 

43 

Midwest 

168 

40 

Northeast and Islands 

140 

37 

Northwest 

115 

30 

Pacific 

— 

— 

Southeast 

179 

39 

Southwest 

163 

37 

West 

151 

38 


—Not applicable 

:f Reporting standards were not met. 

Source: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


How familiar are state and district administrators with the REL program? 

Eighty-six percent of state administrators and 52 percent of district administrators reported 
being atleast “ a little familiar” with the REL program (Table 3-5). 

■ Fourteen percent of state administrators and 48 percent of district administrators 
reported that they were “not familiar at all” with the REL program (Table 3-5). 
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Table 3-5. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all" with the REL program 
overall— National: School year 2011-12 



Familiarity with the REL program 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

79 

27 

170 

4 

Somewhat familiar 

102 

35 

738 

19 

A little familiar 

68 

23 

1,075 

28 

Not familiar at all 

41 

14 

1,722 

48 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


In all 10 regions, more than 70 percent of state administrators reported being at least “ a little 
familiar” with the REL program (Figure 3-5). 

■ In all 10 regions, more than 70 percent of state administrators reported being at least “a 
little familiar” with the REL program, ranging from a high of 96 percent in the 
Northwest region to a low of 71 percent in the Southwest region. (Figure 3-5). 

■ The percentage of district administrators who reported that they were at least “ a little 
familiar” with the REL program ranged from a high of 68 percent in the Northwest 
region to a low of 44 percent in the Southwest region (Figure 3-5). REL Southwest was 
a first-time REL grantee in FY 2006. 
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Figure 3-5. Percentage of all state and district administrators who reported they were at least 
“a little familiar” with the REL program, by region: School year 2011-12 



Percent □ State □ District 



Region 


NOTE: The total N for state administrators was 290, and the total N for district administrators was 3,705. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


How many state and district administrators use REL services? 


Seventy-seven percent of state administrators and 46 percent of district administrators who 
were at least'''' a little familiar” with the REL program reported that they used one or more 
REL services in the past 12 months. 21 

■ The percentage of state administrators at least “a little familiar” with the REL program 
who reported that they used REL services in the past 1 2 months ranged from 47 
percent in Mid-Atlantic region to 94 percent in the West region (Table 3-6). 

■ The percentage of district administrators at least “a little familiar” with the REL program 
who reported that they used REL services in the past 12 months ranged from 38 
percent in the Southwest region to 62 percent in the West region (Table 3-6). 


21 Unless otherwise specified, the term ‘ at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” and “a little familiar.” Note that state and district administrators’ familiarity with the REL 
program varied across regions, as shown in figure 3-5. 
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Table 3-6. Percentage of state and district administrators at least “a little familiar” with the 

REL program who had used one or more REL services in the past 12 months: School 
year 2011-2012 



Region 

n 

% of State administrators 

National 

191 

77 

Appalachia 

17 

69 

Central 

26 

79 

Mid-Atlantic 

10 

47 

Midwest 

31 

83 

Northeast and Islands 

27 

93 

Northwest 

23 

82 

Pacific 

8 

70 

Southeast 

24 

79 

Southwest 

8 

59 

West 

17 

94 

Region 

n 

% of District administrators 

National 

915 

46 

Appalachia 

95 

41 

Central 

103 

43 

Mid-Atlantic 

109 

48 

Midwest 

82 

41 

Northeast and Islands 

82 

43 

Northwest 

139 

55 

Pacific 

- 

- 

Southeast 

98 

45 

Southwest 

73 

38 

West 

134 

62 


— Not applicable 


NOTE: Administrators’ use of services was contingent on familiarity, which differed considerably for states and districts (Table 3-6). 

Source: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ More than 40 percent of state administrators who were at least “a little familiar” with the 
REL program reported that they used each major type of REL service: technical 
assistance (55 percent), a live or virtual event (46 percent), responses to data or research 
requests (45 percent), and information on the REL’s website (41 percent). Of district 
administrators who were at least “a little familiar” with the REL program, 34 percent 
reported obtaining information from the REL’s website; 20 percent reported attending a 
live or virtual event; 1 3 percent reported receiving a response to a data or research 
request; and 1 1 percent reported receiving technical assistance (Table 3-7). 
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Table 3-7. Percentage of state and district administrators at least “a little familiar” with the 
REL program who used various REL services in the past 12 months— National: 
School year 2011-12 



REL services 

n 

% of State administrators 

A live or virtual event 

116 

46 

Technical assistance 

135 

55 

Responses to data or research requests 

113 

45 

Information on the REL’s website 

103 

41 

REL services 

n 

% of District administrators 

A live or virtual event 

389 

20 

Technical assistance 

216 

li 

Responses to data or research requests 

262 

13 

Information on the REL’s website 

683 

34 


NOTE: The total N for state administrators was 249, and the total N for district administrators was 1,987. Administrators’ use of services 
was contingent on familiarity, which differed considerably for states and districts. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ Among administrators who were at least “a little familiar” with the REL program, 1 9 
percent of state administrators and 33 percent of district administrators reported that 
they did not use any REL services in the past 12 months. When asked why they had not 
used any REL services in the past year, the most common response for state 
administrators (54 percent) was that their needs were met elsewhere. Forty-three 
percent of these state administrators said they did not know what services were available 
from the RELs. The most common response for district administrators was that they 
did not know what services were available (58 percent) (Table 3-8). 

Table 3-8. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— National: School year 2011-12 


Reason 

State administrators 

n 

% 

Needs were met elsewhere 

25 

54 

Didn’t know what services were available 

21 

43 

Had no need for REL resources 

12 

24 

Not a good match between their current needs and the REL’s resources 

8 

16 

REL that served their state did not have a good reputation 

t 

t 

Reason 

District administrators 

n 

% 

Needs were met elsewhere 

117 

56 

Didn’t know what services were available 

368 

58 

Had no need for REL resources 

117 

18 

Not a good match between their current needs and the REL’s resources 

51 

8 

REL that served their state did not have a good reputation 

10 

2 


NOTE: The total N for state administrators was 47, and the total N for district administrators was 647. Administrators’ use of services 
was contingent on familiarity, which differed considerably for states and districts. 


t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 
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■ In addition to asking respondents about the services they had used, state and district 
administrators who had used REL services were also asked about the types of contact 
they had in the past 1 2 months with the REL that served their region: 

— Majorities of the state administrators who were at least “a little familiar” with the 
REL program and used REL services indicated that they or their organization 
contacted the REL for research or other assistance (76 percent); they had a REL 
representative attend a meeting or workshop the administrator also attended (74 
percent); or they had attended a REL-sponsored conference, training, or 
workshop (54 percent) (Table 3-9). 

— More than 30 percent of district administrators who were at least “a little familiar” 
with the REL program and had used REL services said a REL representative was 
present at a meeting the administrator also attended (36 percent); they attended a 
REL-sponsored conference, training, or workshop (34 percent); they or their 
organization contacted the REL for research or other assistance (32 percent); or 
they contacted the reference desk for help or used the Ask a REL link on the 
website (32 percent) (Table 3-9). 


Table 3-9. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— National: 
School year 2011-12 


Contact 

State 

District 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link on the website 

36 

19 

260 

32 

Attended a REL-sponsored conference, training, or workshop 

101 

54 

274 

34 

A REL representative was present at a meeting or workshop 

138 

74 

286 

36 

Contacted REL for research or other assistance 

142 

76 

257 

32 

Forwarded a request to the REL 

43 

23 

90 

12 

Other type of contact 

18 

10 

124 

15 


NOTE: The total N state administrators was 189, and the total N for district administrators was 866. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


How satisfied with the REL program are state and district administrators? 

One-half (50 percent) of state administrators and 26 percent of district administrators who 
were atleast “ a little familiar” with the REL program were “very satisfied” with it. 

Of the state administrators who reported being at least “a little familiar” with the REL program, 50 
percent reported being “very satisfied” with it; 48 percent reported being “somewhat satisfied” with 
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it; and 3 percent reported being “not at all satisfied” with it (Figure 3-6). The level of satisfaction 
varied by familiarity with the REL program (Figure 3-7). 

■ Of the district administrators who reported being at least “a little familiar” with the REL 
program, 26 percent reported being “very satisfied” with it; 58 percent reported being 
“somewhat satisfied” with it; and 16 percent report being “not at all satisfied” with it 
(Figure 3-6). 

• Of state administrators who reported being “very familiar” with the REL program, 70 
percent were “very satisfied” with it, while 49 percent of those who reported being 
“somewhat familiar” and 20 percent of those who reported being “a little familiar” with 
the REL program were “very satisfied” with it. Of the district administrators who 
reported being “very familiar” with the REL program, 65 percent were “very satisfied” 
with it compared to 31 percent of the district administrators who reported being 
“somewhat familiar” with REL program and 13 percent who reported being “a little 
familiar” with it (Figure 3-7). Level of familiarity and satisfaction may also vary by other 
factors that were not examined, such as use of services. 

Figure 3-6. Percentage of administrators who reported that they were “very familiar,” 

“somewhat familiar,” or “a little familiar” with the REL program and were “very 
satisfied,” or “somewhat or not at all satisfied” with it— National: School year 
2011-12 



Percent 

100 % 

80% 

60% 

40% 

20% 

0 % 



State 


District 


NOTE: The total N for state administrators was 216, and the total N for district administrators was 1,619. Percentages may not sum to 
100 due to rounding. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 
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Figure 3-7. Percentage of administrators who reported that they were “very familiar,” 

“somewhat familiar,” or “a little familiar” with the REL program and were “very 
satisfied,” or “somewhat or not at all satisfied” with it— National: School year 2011- 
12 



□ Very satisfied ■ Somewhat or not at all satisfied 



State 


Level of familiarity 


District 


NOTE: The total N for state administrators was 216, and the total N for district administrators was 1,618. Percentages may not sum to 
100 due to rounding. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


■ Across the 10 regions served by the REL program, the percentage of state 
administrators who were at least “a little familiar” with the REL program and reported 
being “very satisfied” with it ranged from a high of 65 percent in the Southeast region 
to a low of 28 percent in the Mid-Atlantic region (Table 3-10). 

■ The percentage of district administrators who were at least “a little familiar” with the 
REL program and reported being “very satisfied” with it ranged from a high of 34 
percent in the West region to a low of 17 percent in the Midwest region (Table 3-10). 
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Table 3-10. Percentage of state and district administrators who were at least “a little familiar” 
with the REL program who were “very satisfied” with it, by region: School year 
2011-12 



Region 

n 

% of State administrators 

National 

107 

50 

Appalachia 

11 

61 

Central 

11 

39 

Mid-Atlantic 

4 

28 

Midwest 

19 

56 

Northeast and Islands 

18 

63 

Northwest 

11 

39 

Pacific 

t 

* 

Southeast 

17 

65 

Southwest 

5 

48 

West 

10 

61 

Region 

n 

% of District administrators 

National 

422 

26 

Appalachia 

32 

18 

Central 

54 

27 

Mid-Atlantic 

47 

26 

Midwest 

27 

17 

Northeast and Islands 

40 

28 

Northwest 

68 

29 

Pacific 

- 

- 

Southeast 

50 

29 

Southwest 

39 

26 

West 

65 

34 


— Not applicable 


$ Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011-spring 2012. 


Summary 

To answer the research question “What were the technical quality and relevance of the REL impact 
study reports published by IES and corresponding proposals?” the study team organized an expert 
panel review of the 1 1 impact study proposals and 8 impact study reports published by IES between 
November 2009 and April 2011. The expert panelists rated the quality of the impact study reports 
as, on average, between “strong” and “very strong” and the quality of the impact study proposals as 
between “adequate” and “strong.” For relevance, the expert panelists rated the impact study reports 
as between “relevant” and “very relevant” and the impact study proposals as between “adequate” 
and “relevant.” 
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Cross-REL Findings on the Quality, Relevance, 
and Usefulness of Research and Technical 
Assistance Products 

The study team surveyed state and district administrators to answer the second research question 
“How relevant and useful were the REL technical assistance products to the needs of states and 
districts in their region?” First, administrators were asked about their needs for education research 
and technical assistance and whether those needs were met. Twenty-nine percent of state 
administrators and 26 percent of district administrators reported that their research and technical 
assistance needs were met “very well,” regardless of the source. When asked how much they relied 
on the REL program for education research and/ or technical assistance, 49 percent of state 
administrators and 1 8 percent of district administrators reported that they relied on the REL 
program “to a great extent” or “to a moderate extent.” 

Administrators were then asked about their familiarity and satisfaction with the REL program. 
Eighty-six percent of state administrators and 52 percent of district administrators were at least “a 
little familiar” with the REL program. Of administrators who were at least “a little familiar” with the 
REL program, 77 percent of state administrators and 46 percent of district administrators reported 
that they used one or more REL services in the past 12 months. For administrators who had not 
used any REL services in the past 1 2 months, when asked why they had not used any REL services 
the most common response for state administrators was that their needs were met elsewhere (54 
percent) and for district administrators was that they did not know what services were available (58 
percent). Finally, 50 percent of state administrators and 26 percent of district administrators who 
were at least “a little familiar” with the REL program were “very satisfied” with it. 
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REL: Appalachia 




Regional Educational Laboratory Appalachia serves the following states: 

■ Virginia; 

■ West Virginia; 

■ Tennessee; and 

■ Kentucky. 

For the 2006-1 1 contract period, REL Appalachia was headquartered at the CNA Institute for 
Public Research, a nonprofit research firm in Alexandria, Virginia. The FY 2006 award was the first 
time CNA had held a REL contract, and CNA was also awarded the REL Appalachia contract 
beginning in FY 2012. 


What were the technical quality and relevance of REL Appalachia impact study 
reports published by IES and of the corresponding proposals ? 22 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 2011, IES did not publish any impact studies from REL 
Appalachia, but one proposal was reviewed as part of the evaluation: 

■ Effects of the Kentucky Virtual Schools’ Hybrid Program for Algebra I on Grade 9 Student Math 

Achievement. 25 


22 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using 
randomized controlled trials or regression discontinuity designs. 

23 This study was published by IES in April 2012. 
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The Kentucky Virtual Schools hybrid algebra I curriculum uses online resources in face-to-face 
technology-enhanced classrooms to facilitate the use of standards-based instructional practices. 
Participating teachers engage in sustained professional development focusing on effective pedagogy 
and the use of technology. This study tested whether the program improves classroom quality and 
increases student engagement and achievement. As reported in the study, researchers found that the 
hybrid class format was no more effective at increasing student achievement and future coursetaking 
in math than algebra offered in the traditional face-to-face format. 

The average quality rating for the impact study proposal from REL Appalachia that was 
reviewed by the expert panel was 3.56. On a 5-point scale (1-5, with 1 being the lowest rating 
and 5 the highest), the average proposal quality rating fell between “adequate” and “strong” 
(Table 4-1). 

Table 4-1. Expert panel quality and relevance ratings for impact study proposals from REL 
Appalachia (on a 5-point scale with 5 being the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

3.56 

2.80 


Table Reads: For the impact study proposal for Effects of the Kentucky Virtual Schools’ Hybrid Program for Algebra I on Grade 9 Student 
Math Achievement, the mean quality dimension rating was 3.56. 


NOTE: The mean quality rating for proposals for REL Appalachia was based on 27 indicator-specific ratings, and the mean relevance 
rating for proposals for REL Appalachia was based on 15 indicator-specific ratings. 

SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 

The average relevance rating for the impact study proposal from REL Appalachia that was 
reviewed by the expert panel was 2.80. On a 5-point scale (1-5, with 1 being the lowest rating 
and 5 the highest), the average proposal relevance rating fell between “marginally relevant” 
and “adequate” (Table 4-1). 

Table 4-2 displays, for each indicator of quality and relevance, the mean rating from the expert panel 
review of the proposal from REL Appalachia. 
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Table 4-2. 


REL: Appalachia 



Mean ratings from expert panel review of the impact study proposal for REL 
Appalachia, by rating indicator 


Indicators for proposals 

Proposals 

(/v=D 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

4.33 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

3.33 

ID. The design for random assignment is rigorous. 

3.33 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.67 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.00 

1G. The data collection plan is appropriate for the research questions. 

3.67 

1H. The data analyses will use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

3.33 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

4.00 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to meet 
the needs of the region served by the REL. 

2.67 

2B. The proposal provides a thorough summary of key literature and/or previous research in 
the topic area. 

2.67 

2C. The proposal provides a strong justification for selecting the particular intervention that is 
being studied. 

3.00 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

3.00 

2E. The proposal is clearly written and well presented. 

2.67 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


How relevant and useful were the REL Appalachia technical assistance products 
to the needs of the states and districts in the region? 


Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Appalachia technical assistance products were 
in meeting the needs of administrators in the Appalachia region. State and district administrators 
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were included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Appalachia. -4 



What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Appalachia region was high school reform 
(45%). The most commonly reported area of “high need” for education research and/or 
technical assistance among district administrators in the Appalachia region was 
achievement gaps (46%). 

Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the six largest 
percentages of state administrators in the Appalachia region indicated a “high need” were: 

■ High school reform (45%); 

■ Achievement gaps (43%); 

■ College or career readiness (39%); 

■ Dropout prevention (39%); 


24 Results for the nation are presented in Chapter 3. 
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■ Professional development (38%); and 

■ Rural schools (38%). 

In the Appalachia region, the five topic areas with the largest percentage of district administrators 
reporting a “high need” were: 

■ Achievement gaps (46%); 

■ Content standards, curriculum or instruction in science, technology, engineering or 
mathematics (41%); 

■ College or career readiness (37%); 

■ Using data for decisionmaking (36%); and 

■ Content standards, curriculum or instmction in: reading/writing (35%). 

Detail on the need for other areas of research and technical assistance is provided in Table 4-3. 



Twenty-three percent of state administrators and 27 percent of district administrators in the 
Appalachia region reported that their education research and technical assistance needs 
were met “very well” (as opposed to “moderately well” or “not well”). 

■ Seventy-seven percent of state administrators in the Appalachia region reported that 
their education research and technical assistance needs were met “moderately well” and 
0 percent reported that their needs were “not well” met by their sources of 
assistance(Table 4-4). 

■ Sixty-six percent of district administrators in the Appalachia region reported that their 
education research and technical assistance needs were met “moderately well” and 7 
percent reported that their needs were “not well” met by their sources of assistance 
(Table 4-4). 
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Table 4-3. 


Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Appalachia: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

26 

43 

41 

16 

421 

46 

41 

12 

Assessment (formative or summative) 

26 

17 

57 

26 

420 

30 

49 

21 

Behavior, character education, or health 

26 

11 

35 

54 

417 

15 

44 

41 

College or career readiness 

26 

39 

45 

16 

418 

37 

42 

21 

Content standards, curriculum or instruction in STEM 

26 

26 

32 

42 

420 

41 

42 

17 

Content standards, curriculum or instruction in reading/writing 

26 

26 

24 

50 

419 

35 

48 

18 

Content standards, curriculum or instruction in other areas 

26 

t 

t 

58 

415 

14 

48 

39 

Dropout prevention 

26 

39 

37 

24 

419 

33 

40 

27 

Early childhood 

26 

31 

34 

35 

418 

21 

42 

37 

English language learners 

26 

30 

47 

23 

418 

19 

37 

44 

High school reform 

26 

45 

14 

41 

416 

24 

44 

32 

Leadership 

27 

30 

51 

18 

419 

21 

54 

25 

Longitudinal data systems 

26 

20 

42 

38 

418 

24 

48 

28 

Parental involvement 

26 

27 

52 

21 

421 

26 

48 

25 

Professional development 

26 

38 

51 

11 

419 

26 

51 

23 

Rural schools 

26 

38 

40 

22 

418 

22 

32 

45 

School accountability 

26 

24 

38 

38 

419 

17 

49 

33 

School choice 

26 

t 

t 

62 

417 

3 

24 

73 

School finance 

26 

t 

t 

62 

416 

12 

35 

53 

Students with disabilities 

26 

24 

45 

32 

416 

29 

49 

22 

Supplemental education services 

26 

t 

t 

51 

418 

10 

43 

47 

Support for low-achieving schools 

26 

26 

59 

15 

420 

34 

39 

27 

Teacher/staff evaluation 

27 

34 

40 

25 

418 

28 

45 

27 

Using data for decisions 

27 

28 

61 

10 

418 

36 

44 

21 

Other 

3 

t 

t 

t 

51 

38 

8 

54 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Table 4-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well," or “not well," taking into 
account all sources of assistance— Appalachia: School year 2011-12 



How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

6 

23 

114 

27 

Moderately well 

19 

77 

275 

66 

Not well 

0 

0 

30 

7 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported source of education research and/or technical assistance for state 
administrators was professional associations (90%), and for district administrators it was 
counterparts in other states and districts (85%). Forty-two percent of state administrators 
and 18 percent of district administrators in the Appalachia region reported that they relied 
on the REL program “to a great extent” or “to a moderate extent” for research and/ or 
technical assistance. 

■ State and district administrators in the Appalachia region reported that they used a 
variety of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on professional associations (90%) and other 
products or resources from ED (82%), while district administrators were most likely to 
rely “to a great extent” or “to a moderate extent” on counterparts in other states and 
districts (85%) and journals and publications (73%) (Figure 4-1). 

■ Forty-two percent of state administrators and 18 percent of district administrators in the 
Appalachia region reported that they relied on the REL program overall “to a great 
extent” or “to a moderate extent” for research and/or technical assistance (Figure 4-1). 

■ Thirty percent of state administrators in the Appalachia region reported that it was 
“very easy” (as opposed to “moderately easy” or “not at all easy”) to access education 
research and/or technical assistance across the available sources of information, and 38 
percent of district administrators in the Appalachia region reported that it was “very 
easy” to access such assistance (Table 4-5). 
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Figure 4-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Appalachia: School year 2011-12 


REL program nationwide 
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NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent" on other sources of education research and/or technical 
assistance. The total N for state administrators was 23. The total N for district administrators on the item about reliance on the REL 
program was 224, and the total N for district administrators on the items about all other specified sources of research ranged from 
419 to 422, depending on the number of district respondents who chose not to respond to an individual item. The total N for district 
administrators for “other sources" was 22. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Table 4-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Appalachia: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

8 

30 

160 

38 

Moderately easy to access 

t 

t 

239 

57 

Not at all easy to access 

t 

t 

21 

5 


NOTE: if Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional ^Jucational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How familiar were state and district administrators with the REL program? 


Ninety percent of state administrators in the Appalachia region reported being at least “a 
little familiar” with the REL program, compared with 55 percent of district administrators. 


■ Ten percent of state administrators and 45 percent of district administrators in the 

Appalachia region reported that they were “not familiar at all” with the REL program 
(Table 4-6). 

Table 4-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all" with the REL program 
overall— Appalachia: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

5 

18 

13 

3 

Somewhat familiar 

12 

46 

77 

18 

A little familiar 

7 

26 

142 

34 

Not familiar at all 

3 

10 

189 

45 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How many state and district administrators used REL services? 

Sixty-nine percent of state administrators and 41 percent of district administrators in the 
Appalachia region who were at least “a little familiar” with the REL program reported that 
they used one or more REL services in the past 12 months. 25 

■ The majority of state administrators in the Appalachia region who were at least “a little 
familiar” with the REL program reported that they used REL technical assistance 
services in the past 12 months (61%). Of district administrators in the Appalachia 
region who were at least “a little familiar” with the REL program, 32 percent obtained 
information from the REL’s website; 1 5 percent attended a live or virtual event; 1 2 
percent received a response from a data or research request; and 7 percent received 
technical assistance (Figure 4-2). 


25 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Appalachia region. 
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Figure 4-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Appalachia: School year 2011-12 
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NOTE: The total N for state administrators was 23; the total N for district administrators was 168. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Of administrators in the Appalachia region who were at least “a little familiar” with the 
REL program, 27 percent of state administrators and 32 percent of district 
administrators did not use any REL services in the past 12 months. 26 When asked why 
they had not used any REL services in the past year, the most common responses for 
state and district administrators were that they didn’t know what services were available 
(64 and 55 percent, respectively), and their needs were met elsewhere (53 and 52 
percent, respectively) (Table 4-7). 


26 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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REL: Appalachia 

Reasons administrators who were at least “a little familiar” with the REL Program 
did not use REL services in the past 12 months— Appalachia: School year 2011-12 


Reason 

State 

District 

n 

% 

n 

% 

Needs were met elsewhere 

3 

53 

30 

52 

Didn’t know what services were available 

4 

64 

27 

55 

Had no need for REL resources 

0 

0 

17 

15 

Not a good match between their current needs and the REL’s resources 

0 

0 

t 

t 

REL that served their state did not have a good reputation 

0 

0 

0 

0 


NOTE: The total N for state administrators was 6, and the total N for district administrators was 73. t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Appalachia region who had used REL services were also asked about the types of contact they 
had in the past 1 2 months with the REL. 


■ Majorities of state administrators in the Appalachia region who were at least “a little 

familiar” with the REL program and had used REL Appalachia services in the past year 
indicated that they attended a meeting or workshop at which a REL representative was 
present (74%), or their organization contacted the REL for research or other assistance 
(73%). Among district administrators in the Appalachia region who were at least “ a little 
familiar” with the REL program and had used REL Appalachia services in the past year, 
39 percent said they attended a meeting at which a REL representative was present; 31 
percent contacted a reference desk for help or used the Ask a REL link on the REL’s 
website; 30 percent said they attended a REL-sponsored conference, training, or 
workshop; 24 percent said that they or their organization contacted the REL for 
research or other assistance; 10 percent said they forwarded someone else’s request to 
the REL; and 16 percent had some other type of contact (Table 4-8). 

Table 4-8. Percentage of administrators who had used REL services and reported having 
various types of contact with the REL serving their state in the past year— 
Appalachia: School year 2011-12 


Contact 

State 

District 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link on the website 

3 

20 

25 

31 

Attended a REL-sponsored conference, training, or workshop 

7 

47 

24 

30 

A REL representative was present at a meeting or workshop 

11 

74 

31 

39 

Contacted REL for research or other assistance 

11 

73 

19 

24 

Forwarded a request to the REL 

t 

* 

8 

10 

Other type of contact 

t 

t 

13 

16 


NOTE: The total N for state administrators was 17, and the total N for district administrators was 91. t Reporting standards were not 
met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How satisfied with the REL program were state and district administrators? 


Sixty-one percent of state administrators and 18 percent of district administrators in the 
Appalachia region who were atleast u a little familiar” with the REL program overall were 
“very satisfied” with it. 


■ Of the district administrators in the Appalachia region who were at least “a little 
familiar” with the REL program overall, 67 percent reported being “somewhat 
satisfied” with it and 16 percent reported being “not at all satisfied” with it (Table 4-9). 

Table 4-9. Percentage of the region’s administrators at least “a little familiar” with the REL 
program who were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” 
with it— Appalachia: School year 2011-12 


Satisfaction 

State 

District 

n 

% 

n 

% 

Very satisfied 

11 

61 

32 

18 

Somewhat satisfied 

t 

t 

122 

67 

Not at all satisfied 

$ 

t 

29 

16 


NOTE: Percentages may not sum to 100 due to rounding, if Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Regional Educational Laboratory Central serves the following states: 

■ Colorado; 

■ Kansas; 

■ Missouri; 

■ Nebraska; 

■ North Dakota; 

■ South Dakota; and 

■ Wyoming. 

For the 2006-1 1 contract period, REL Central was housed at Mid-continent Research for Education 
and Learning (McREL). At the time of data collection, McREL had held the REL contract 
continuously since 1966. Marzano Research Laboratory was awarded the REL Central contract 
beginning in FY 2012. 


What were the technical quality and relevance of REL Central impact study 
reports published by IES and of the corresponding proposals ? 27 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 2011, IES published one impact study from REL Central: 


27 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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5 


■ Classroom Assessment for Student Learning: Impact on Llementary School Mathematics in the 

Central Legion. 

C4SL is a professional development program on classroom and formative assessment that includes 
a textbook, DVDs, ancillary books, and an implementation handbook, all of which are used to train 
teachers to conduct classroom assessments that are appropriate for and aligned with their learning 
targets. The study used a random assignment design to investigate the effectiveness of the Classroom 
Assessment for Student Learning fCASLJ program by comparing mathematics achievement of 
elementary school students in the CASL and comparison groups in the spring of the 
implementation year. The study found no effects of CASL on the mathematics achievement of 
fourth- and fifth-grade students. 


The average quality ratings for the impact study report and proposal from REL Central that 
were reviewed by the expert panel were 4.11 and 3.96, respectively. On a 5-point scale (1-5, 
with 1 being the lowest rating and 5 the highest), the average report quality rating fell 
between “strong” and “very strong,” and the average proposal quality rating fell between 
“adequate” and “strong” (Table 5-1). 

Table 5-1. Expert panel quality and relevance ratings for lES-published impact study reports 
and corresponding proposals from REL Central (on a 5-point scale with 5 being the 
highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

3.96 

3.73 

lES-published impact study report 

4.11 

3.72 


Table Reads: For the proposal Classroom Assessment for Student Learning: Impact on Elementary School Mathematics in the Central 
Region, the mean quality dimension rating was 3.96. 


NOTE: The mean quality rating for proposals for REL Central was based on 27 indicator-specific ratings, and the mean relevance rating 
for proposals for REL Central was based on 15 indicator-specific ratings. The mean quality rating for reports for REL Central was based 
on 45 indicator-specific ratings, and the mean relevance rating for reports for REL Central was based on 18 indicator-specific ratings. 
SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


The average relevance ratings for the impact study report and proposal from REL Central 
that were reviewed by the expert panel were 3.72 and 3.73, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report and proposal 
relevance ratings fell between “adequate” and “relevant” (Table 5-1). 

Table 5-2 displays, for each indicator of quality and relevance, the mean ratings from expert panel 
reviews of the IES-published impact study and corresponding proposal from REL Central. 
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Table 5-2. Mean ratings from expert panel review of impact studies for REL Central, by rating 
indicator 


Indicators for proposals 

Proposals 

(/v=D 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

4.00 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

4.00 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

4.00 

ID. The design for random assignment is rigorous. 

4.00 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

4.33 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.00 

1G. The data collection plan is appropriate for the research questions. 

4.33 

1H. The data analyses will use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

3.33 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

4.67 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

3.33 

2B. The proposal provides a thorough summary of key literature and/or previous research in 
the topic area. 

3.00 

2C. The proposal provides a strong justification for selecting the particular intervention that 
is being studied. 

3.33 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

5.00 

2E. The proposal is clearly written and well presented. 

4.00 
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Table 5-2. Mean ratings from expert panel review of impact studies for REL Central, by rating 
indicator (continued) 


Indicators for reports 

lES-published 
reports 
(N = 1) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.67 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

4.00 

1C. There is minimal contamination in the form of crossover between subjects in 

treatment and control condition or spillover of the intervention from the treatment 
to the control group. 

4.67 

ID. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.00 

IE. The design and implementation of random assignment are rigorous. 

4.00 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

3.33 

1G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 

3.67 

1H. The data collection plan is appropriate for the research questions. 

3.67 

11. The data collection plan is well implemented. 

4.67 

U. The overall attrition rate and differential attrition rates are acceptable given the 
length of the intervention. 

3.33 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

4.00 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.33 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

4.67 

IN. All of the research questions are specifically addressed by the analyses. 

5.00 

10. The limitations of the study are clearly and comprehensively stated. 

4.67 

Relevance 

2A. The report provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.00 

2B. The report provides a thorough summary of key literature and/or previous research 
in the topic area. 

3.00 

2C. The report provides a strong justification for selecting the particular intervention that 
is being studied. 

3.33 

2D. The report contributes new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.33 

2E. The executive summary of the report is easy to read and understand for a lay 
audience. 

3.33 

2F. The report is clear and well written for the technical audience. 

4.33 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


Source: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports); 
Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals) 
(Appendix A). 
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How relevant and useful were the REL Central technical assistance products to 
the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Central technical assistance products were in 
meeting the needs of administrators in the Central region. State and district administrators were 
included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL Program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL Program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Central. 2S 

What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Central region was support for low-achieving 
schools (67%). The most commonly reported area of “high need” for education research 
and/or technical assistance among district administrators in the Central region was content 
standards, curriculum, or instruction in areas other than reading/writing or STEM (36%). 


28 Results for the nation are presented in chapter 3. 
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Administrators were asked to indicate whether they had a “high need, “moderate need,” or “low or 
no” need for research/ assistance in specific topic areas. The areas in which the five largest 
percentages of state administrators in the Central region indicated “high need” were: 

■ Support for low-achieving schools (67%); 

■ Rural schools (63%); 

■ Achievement gaps (59%); 

■ College or career readiness (54%); and 

■ Teacher/ staff evaluation (52%). 

In the Central region, the five topic areas with the largest percentage of district administrators 
reporting “high need” were: 

■ Content standards, curriculum, or instmction in areas other than reading/ writing, or 
STEM (36%); 

■ Content standards, curriculum or instmction in reading/ writing (31%); 

■ Using data for decisions (30%); 

■ Assessment (29%); and 

■ Achievement gaps (28%). 

Detail on the need for other areas of research and technical assistance is provided in Table 5-3. 
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Table 5-3. 


Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Central: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or 
no need 


High 

Moderate 

Low or 
no need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

39 

59 

31 

11 

408 

28 

52 

21 

Assessment (formative or summative) 

39 

29 

55 

16 

408 

29 

47 

24 

Behavior, character education, or health 

39 

13 

46 

42 

406 

18 

46 

36 

College or career readiness 

39 

54 

30 

16 

408 

24 

46 

30 

Content standards, curriculum or instruction in STEM 

38 

32 

48 

19 

407 

36 

46 

19 

Content standards, curriculum or instruction in reading/writing 

38 

32 

45 

24 

406 

31 

51 

18 

Content standards, curriculum or instruction in other areas 

38 

21 

35 

44 

403 

7 

50 

43 

Dropout prevention 

38 

26 

t 

t 

407 

22 

39 

39 

Early childhood 

38 

48 

39 

13 

408 

23 

39 

38 

English language learners 

38 

47 

46 

7 

408 

17 

35 

48 

High school reform 

38 

28 

59 

13 

408 

16 

45 

40 

Leadership 

38 

40 

52 

7 

406 

21 

51 

28 

Longitudinal data systems 

38 

34 

42 

24 

408 

21 

50 

28 

Parental involvement 

38 

34 

54 

13 

408 

27 

47 

26 

Professional development 

38 

38 

47 

16 

408 

24 

55 

21 

Rural schools 

38 

63 

26 

11 

407 

24 

33 

43 

School accountability 

38 

48 

33 

19 

408 

12 

53 

35 

School choice 

38 

7 

23 

69 

408 

4 

23 

73 

School finance 

38 

10 

39 

51 

408 

16 

41 

44 

Students with disabilities 

38 

21 

55 

24 

408 

18 

50 

32 

Supplemental education services 

38 

24 

34 

42 

408 

9 

44 

47 

Support for low-achieving schools 

38 

67 

22 

12 

407 

22 

40 

37 

Teacher/staff evaluation 

38 

52 

40 

8 

408 

24 

47 

28 

Using data for decisions 

38 

46 

t 

t 

407 

30 

46 

24 

Other 

t 

t 

t 

t 

34 

27 

t 

t 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. 4 Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Thirty-five percent of state administrators and 26 percent of district administrators in the 
Central region reported that their education research and technical assistance needs were 
met “very well” (as opposed to “moderately well” or “not well”), taking into account all 
sources of such research and technical assistance. 


■ Sixty-six percent of district administrators in the Central region reported that their 

education research and technical assistance needs were met “moderately well” by their 
sources of assistance and 8 percent reported that their needs were “not well” met (Table 
5-4). 

Table 5-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Central: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

13 

35 

106 

26 

Moderately well 

X 

X 

268 

66 

Not well 

X 

* 

33 

8 


NOTE: t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported source of education research and/ or technical assistance for state 
administrators was professional associations (100%). For district administrators, the most 
reported sources were education journals and publications (69%) and counterparts in other 
states and districts (69%). Fifty-four percent of state administrators and 19 percent of district 
administrators in the Central region reported that they relied on the REL program “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance. 

■ State and district administrators in the Central region reported that they used a variety 
of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on professional associations (100%) and 
counterparts in other states and districts (82%), while district administrators were most 
likely to rely “to a great extent” or “to a moderate extent” on education journals and 
publications (69%) and counterparts in other states and districts (69%) (Figure 5-1). 
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■ Fifty-four percent of state administrators and 1 9 percent of district administrators in the 
Central region reported that they relied on the REL program nationwide “to a great 
extent” or “to a moderate extent” for research and/or technical assistance (Figure 5-1). 

Figure 5-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Central: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from 
ED, including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on “other sources" of education research and/or technical 
assistance. The total N for state administrators was 39. The total N for district administrators on the item about reliance on the REL 
program was 409; the total N for district administrators on the items about other specified sources of research ranged from 407 to 
408, depending on the number of district respondents who chose not to respond to an individual item; and the total N for district 
administrators for “other sources” was 34. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Twenty-nine percent of state administrators in the Central region reported that it was 
“very easy” (as opposed to “moderately easy” or “not at all easy”) to access education 
research and/or technical assistance across the available sources of information, and 40 
percent of district administrators in the Central region reported that it was “very easy” 
to access such assistance (Table 5-5). 
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Table 5-5. Percentage of all administrators who reported that it was ’’very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Central: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

11 

29 

161 

40 

Moderately easy to access 

t 

t 

213 

52 

Not at all easy to access 

t 

t 

35 

9 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Eighty-four percent of state administrators in the Central region reported being at least “a. 
little familiar” with the REL program, compared with 59 percent of district administrators. 

■ Sixteen percent of state administrators and 40 percent of district administrators in the 
Central region reported that they were “not familiar at all” with the REL program 
(Table 5-6). 


Table 5-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all" with the REL program 
overall —Central: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

10 

25 

18 

4 

Somewhat familiar 

20 

51 

91 

22 

A little familiar 

3 

8 

134 

33 

Not familiar at all 

6 

16 

165 

40 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How many state and district administrators used REL services? 

Seventy-nine percent of state administrators and 43 percent of district administrators in the 
Central region who were at least “ a little familiar” with the REL program reported that they 
used one or more REL services in the past 12 months. 29 

■ The majority of state administrators in the Central region who were at least “a little 
familiar” with the REL program reported that they used information on the REL’s 
website in the past 12 months (54%). Of district administrators in the Central region 
who were at least “a little familiar” with the REL program, 32 percent used information 
on the REL’s website; 1 7 percent attended a live or virtual event; 1 2 percent received 
technical assistance from the REL; and 10 percent received a response from a data or 
research request (Figure 5-2). 


Figure 5-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Central: School year 2011-12 


Used one or more services 


A live or virtual event 


Technical assistance 

Responses to data or research 
requests 

Information on the REL's website 



79 


DD State 
□ District 


0 20 40 60 80 100 % 


NOTE: The total N for state administrators was 33, and the total N for district administrators was 244. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


29 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Central region. 
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■ Of administrators in the Central region who were at least “a little familiar” with this REL 
program, 21 percent of state administrators and 36 percent of district administrators did 
not use any REL services in the past 12 months/ 1 " When asked why they had not used 
any REL services in the past year, the most common responses for state administrators 
(57 percent and 43 percent, respectively), were that their needs were met elsewhere or 
they didn’t know what services were available. Likewise, the most common responses 
for district administrators who were at least “a little familiar” with the REL program but 
had not used any REL services in the past 12 months (55 and 56 percent, respectively) 
were that their needs were met elsewhere or they didn’t know what services were 
available (Table 5-7). 

Table 5-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Central: School year 2011-12 


Reason 

State 

District 

n 

% 

n 

% 

Needs were met elsewhere 

4 

57 

48 

55 

Didn’t know what services were available 

3 

43 

49 

56 

Had no need for REL resources 

t 

t 

20 

23 

Not a good match between their current needs and the REL’s resources 

t 

t 

11 

12 

REL that served their state did not have a good reputation 

t 

t 

t 

t 


NOTE: The total N state administrators was 7, and the total N for district administrators was 89. t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Central region who had used REL services were also asked about the types of contact they 
had in the past 12 months with the REL. 

■ Majorities of state administrators in the Central region who were at least “a little 

familiar” with the REL program and had used REL Central services in the past year 
indicated that they attended a meeting or workshop at which a REL representative was 
present (78%), or they or their organization contacted the REL for research or other 
assistance (66%). Among district administrators in the Central region who were at least 
“a little familiar” with the REL program and had used REL Central services in the past 
year, 40 percent said they attended a REL-sponsored conference, training, or workshop; 
39 percent said that they or their organization contacted the REL for research or other 
assistance; 37 percent contacted a reference desk for help or used the “Ask a REL” link 
on the REL’s website; 3 1 percent said they attended a meeting at which a REL 
representative was present; and 1 1 percent forwarded a request they had received to the 
REL (Table 5-8). 


30 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 5-8. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Central: 
School year 2011-12 


Contact 

State 

District 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link on the website 

5 

20 

32 

37 

Attended a REL-sponsored conference, training, or workshop 

11 

43 

58 

40 

A REL representative was present at a meeting or workshop 

20 

78 

54 

31 

Contacted REL for research or other assistance 

17 

66 

45 

39 

Forwarded a request to the REL 

3 

12 

18 

11 

Other type of contact 

t 

t 

22 

10 


NOTE: The total N for state administrators was 26, and the total N for district administrators was 97. $ Reporting standards were not 
met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 


Forty percent of state administrators and 27 percent of district administrators in the Central 
region who were at least “ a little familiar” with the REL program overall were “very 
satisfied” with it. 


■ Of the state administrators in the Central region who were at least “a little familiar” with 
the REL program overall, 61 percent reported being “somewhat satisfied” with it and 0 
percent of state administrators reported being “not at all satisfied” with it (Table 5-9). 


■ Of the district administrators in the Central region who were at least “a little familiar” 

with the REL program overall, 60 percent reported being “somewhat satisfied” with it 
and 13 percent reported being “not at all satisfied” with it (Table 5-9). 

Table 5-9. Percentage of the region’s administrators at least “a little familiar” with the REL 
program who were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” 
with it— Central: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

11 

39 

54 

27 

Somewhat satisfied 

19 

61 

121 

60 

Not at all satisfied 

0 

0 

26 

13 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Regional Educational Laboratory Mid-Atlantic serves the following states: 

■ Delaware; 

■ District of Columbia; 

■ Maryland; 

■ New Jersey; and 

■ Pennsylvania. 

For the 2006-1 1 contract period, REL Central was headquartered at Pennsylvania State University 
(PSU) and included four primary subcontractors: Rutgers University, ICF International, 
ANALYTICA, and the Metiri Group. The 2006-1 1 contract period was the first time PSU or any of 
its partner organizations held a REL contract. The REL Mid-Atlantic contract beginning in FY 2012 
was awarded to ICF International and its partners. 


What were the technical quality and relevance of REL Mid-Atlantic impact study 
reports published by iES and of the corresponding proposals ? 31 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 2011, IES published one impact study from REL Mid- 
Atlantic: 


■ A Multisite Cluster Randomised Trial of the Effects of Compass Teaming Odyssey Math on the 

Math Achievement of Selected Grade 4 Students in the Mid-Atlantic Region. 


31 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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Odyssey® Math , is a web-based K— 8 mathematics curriculum and assessment tool designed to enable 
teachers to differentiate student instruction and make data-driven decisions. The study examined 
whether exposure to Odyssey® Math improved mathematics achievement of fourth-grade students. 

The study found no discernible effects of Odyssey® Math on mathematics achievement in the spring 
of the implementation year. 


The average quality ratings for the impact study report and proposal from REL Mid- 
Atlantic that were reviewed by the expert panel were 4.42 and 4.22, respectively. On a 5- 
point scale (1-5, with 1 being the lowest rating and 5 the highest), the average report and 
proposal quality rating fell between “strong” and “very strong” (Table 6-1). 

Table 6-1. Expert panel quality and relevance ratings for lES-published impact study reports 

and corresponding proposals from REL Mid-Atlantic (on a 5-point scale with 5 being 
the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

4.22 

3.53 

lES-published impact study report 

4.42 

4.39 


Table Reads: For the proposal A Multisite Cluster Randomized Trial of the Effects of Compass Learning Odyssey Math on the Math 
Achievement of Selected Grade 4 Students in the Mid-Atlantic Region, the mean quality dimension rating was 4.22. 


NOTE: The mean quality rating for proposals for REL Mid-Atlantic was based on 27 indicator-specific ratings, and the mean relevance 
rating for proposals for REL Mid-Atlantic was based on 15 indicator-specific ratings. The mean quality rating for reports for REL Mid- 
Atlantic was based on 45 indicator-specific ratings, and the mean relevance rating for reports for REL Mid-Atlantic was based on 18 
indicator-specific ratings. 

SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


The average relevance ratings for the impact study report and proposal from REL Mid- 
Atlantic that were reviewed by the expert panel were 4.39 and 3.53, respectively. On a 5- 
point scale (1-5, with 1 being the lowest rating and 5 the highest), the average report 
relevance rating fell between “relevant” and “very relevant,” and the average proposal 
relevance rating fell between “adequate” and “relevant” (Table 6-1). 

Table 6-2 displays, for each indicator of quality and relevance, the mean ratings from expert panel 
reviews of IES-published impact studies and corresponding proposals from REL Mid-Atlantic. 
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Table 6-2. Mean ratings from expert panel review of impact studies for REL Mid-Atlantic, by 
rating indicator 


Indicators for proposals 

Proposals 

(/v=D 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.67 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

5.00 

1C. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.67 

ID. The design for random assignment is rigorous. 

3.67 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

4.00 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

4.67 

1G. The data collection plan is appropriate for the research questions. 

5.00 

1H. The data analyses will use appropriate methods to address the research questions 
(e.g., adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

3.33 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

4.00 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

3.00 

2B. The proposal provides a thorough summary of key literature and/or previous 
research in the topic area. 

3.67 

2C. The proposal provides a strong justification for selecting the particular intervention 
that is being studied. 

3.00 

2D. The report will contribute new information about the effectiveness of the 
intervention being studied and the more general topic being addressed. 

4.33 

2E. The proposal is clearly written and well presented. 

3.67 
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Table 6-2. Mean ratings from expert panel review of impact studies for REL Mid-Atlantic, by 
rating indicator (continued) 


Indicators for reports 

lES-published 

reports 

(/v=D 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

4.67 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

4.33 

1C. There is minimal contamination in the form of crossover between subjects in 

treatment and control condition or spillover of the intervention from the treatment 
to the control group. 

4.67 

ID. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.33 

IE. The design and implementation of random assignment are rigorous. 

4.33 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

4.00 

1G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 

5.00 

1H. The data collection plan is appropriate for the research questions. 

5.00 

11. The data collection plan is well implemented. 

5.00 

U. The overall attrition rate and differential attrition rates are acceptable given the 
length of the intervention. 

4.67 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

4.00 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.67 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

5.00 

IN. All of the research questions are specifically addressed by the analyses. 

2.33 

10. The limitations of the study are clearly and comprehensively stated. 

4.33 

Relevance 

2A. The report provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.33 

2B. The report provides a thorough summary of key literature and/or previous research 
in the topic area. 

4.00 

2C. The report provides a strong justification for selecting the particular intervention that 
is being studied. 

4.33 

2D. The report contributes new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

5.00 

2E. The executive summary of the report is easy to read and understand for a lay 
audience. 

4.67 

2F. The report is clear and well written for the technical audience. 

4.00 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 
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How relevant and useful were the REL Appalachia technical assistance products 
to the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Mid-Atlantic technical assistance products 
were in meeting the needs of administrators in the Mid-Atlantic region. State and district 
administrators were included in the sample regardless of previous use of REL services or familiarity 
with the REL program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Mid- Atlantic. ’ 2 



What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Mid-Atlantic region was teacher/ staff 
evaluation (53%). The most commonly reported area of “high need” for education research 
and / or technical assistance among district administrators in the Mid- Atlantic region was 
other types of research and/ or technical assistance (41%). 


32 Results for the nation are presented in Chapter 3. 
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Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need for research/ assistance in specific topic areas. The areas in which the five largest 
percentages of state administrators in the Mid-Atlantic region indicated “high need” were: 

■ Teacher/ staff evaluation (53%); 

■ Achievement gaps (45%); 

■ Content standards, curriculum, or instmction in areas other than reading/ writing, or 
STEM (43%); 

■ Assessment (42%); and 

■ Professional development (39%). 

In the Mid-Atlantic region, the five topic areas with the largest percentage of district administrators 
reporting “high need” were: 

■ Other types of research and/ or technical assistance (41%); 

■ Using data for decisions (35%); 

■ Content standards, curriculum, or instmction in areas other than reading/ writing, or 
STEM (33%); 

■ Teacher/ staff evaluation (33%); and 

■ Content standards, curriculum or instmction in reading/writing (31%). 

Detail on the need for other areas of research and technical assistance is provided in Table 6-3. 
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Table 6-3. 


Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance Mid-Atlantic: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

25 

45 

31 

24 

404 

30 

52 

18 

Assessment (formative or summative) 

24 

42 

31 

26 

404 

29 

50 

21 

Behavior, character education, or health 

25 

0 

32 

68 

402 

20 

42 

38 

College or career readiness 

25 

24 

59 

17 

403 

29 

39 

32 

Content standards, curriculum or instruction in STEM 

25 

43 

27 

29 

405 

33 

46 

21 

Content standards, curriculum or instruction in reading/writing 

25 

36 

29 

35 

404 

31 

50 

19 

Content standards, curriculum or instruction in other areas 

25 

20 

34 

46 

400 

10 

49 

41 

Dropout prevention 

25 

37 

28 

35 

403 

23 

33 

44 

Early childhood 

25 

24 

35 

42 

404 

16 

38 

46 

English language learners 

25 

21 

50 

29 

404 

15 

39 

45 

High school reform 

25 

36 

36 

28 

404 

21 

37 

41 

Leadership 

26 

32 

34 

35 

405 

23 

49 

28 

Longitudinal data systems 

25 

26 

31 

43 

405 

29 

45 

26 

Parental involvement 

25 

32 

26 

41 

403 

28 

44 

28 

Professional development 

26 

39 

30 

31 

405 

28 

47 

25 

Rural schools 

25 

* 

X 

39 

403 

12 

21 

67 

School accountability 

25 

36 

44 

20 

404 

16 

45 

39 

School choice 

25 

X 

X 

43 

404 

9 

23 

68 

School finance 

25 

20 

28 

52 

404 

20 

34 

45 

Students with disabilities 

25 

18 

34 

48 

403 

28 

49 

23 

Supplemental education services 

25 

22 

18 

60 

405 

11 

48 

41 

Support for low-achieving schools 

25 

36 

35 

29 

404 

23 

34 

43 

Teacher/staff evaluation 

26 

53 

15 

32 

405 

33 

44 

22 

Using data for decisions 

25 

33 

36 

31 

404 

35 

45 

20 

Other 

0 

0 

0 

0 

44 

41 

7 

52 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. % Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 


o 


REL: Mid-Atlantic 



REL: Mid-Atlantic 


6 


Forty-five percent of state administrators and 26 percent of district administrators in the 
Mid-Atlantic region reported that their education research and technical assistance needs 
were met “very well” (as opposed to “moderately well” or “not well”), taking into account 
all sources of such research and technical assistance. 


■ Sixty-nine percent of district administrators in the Mid-Atlantic region reported that 
their education research and technical assistance needs were met “moderately well” by 
their sources of assistance and 5 percent reported that their needs were “not well” 
met(Table 6-4). 

Table 6-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Mid-Atlantic: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

12 

45 

103 

26 

Moderately well 

t 

t 

279 

69 

Not well 

* 

* 

22 

5 


NOTE: t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance do state and 
district administrators use? 


The most reported source of education research and/ or technical assistance for state administrators 
was professional associations (85%), and for district administrators, it was counterparts in other 
states and districts (85%). Thirty-eight percent of state administrators and 19 percent of district 
administrators in the Mid-Atlantic region reported that they relied on the REL program “to a great 
extent” or “to a moderate extent” for research and/or technical assistance. 

■ State and district administrators in the Mid-Atlantic region reported that they used a 
variety of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on professional associations (85%) and 
counterparts in other states and districts (69%), while district administrators were most 
likely to rely “to a great extent” or “to a moderate extent” on counterparts in other 
states and districts (85%) and journals and publications (83%) (Figure 6-1). 
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■ Thirty-eight percent of state administrators and 1 9 percent of district administrators in 
the Mid-Atlantic region reported that they relied on the REL program nationwide “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance (Figure 
6 - 1 ). 

Figure 6-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Mid-Atlantic: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from 
ED, including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on “other sources" of education research and/or technical 
assistance. The total N for state administrators ranged from 24 to 26, depending on the number of respondents who chose not to 
respond to an individual item. The total N for district administrators on the item about reliance on the REL program was 406; the total 
N for district administrators on the items about other specified sources of research ranged from 403 to 405, depending on the 
number of district respondents who chose not to respond to an individual item; and the total N for district administrators for “other 
sources” was 18. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Forty-four percent of state administrators in the Mid-Atlantic region reported that it 
was “very easy” (as opposed to “moderately easy” or “not at all easy”) to access 
education research and/ or technical assistance across the available sources of 
information, and 43 percent of district administrators in the Mid-Atlantic region 
reported that it was “very easy” to access such assistance (Table 6-5). 
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Table 6-5. Percentage of all administrators who reported that it was ’’very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Mid-Atlantic: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

12 

44 

176 

43 

Moderately easy to access 

X 

X 

210 

52 

Not at all easy to access 

X 

X 

20 

5 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Eighty-five percent of state administrators in the Mid-Atlantic region reported being at least 
“a little familiar” with the REL program, compared with 56 percent of district 
administrators. 


■ Sixteen percent of state administrators and 45 percent of district administrators in the 
Mid-Atlantic region reported that they were ’’not familiar at all” with the REL program 
(Table 6-6). 

Table 6-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or ’’not familiar at all" with the REL program 
overall— Mid-Atlantic: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

6 

24 

17 

4 

Somewhat familiar 

7 

27 

89 

22 

A little familiar 

9 

34 

119 

30 

Not familiar at all 

4 

16 

181 

45 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How many state and district administrators used REL services? 

Forty-seven percent of state administrators and 48 percent of district administrators in the 
Mid-Atlantic region who were at least “a. little familiar” with the REL program reported that 
they used one or more REL services in the past 12 months. 33 

■ Of state administrators in the Mid-Atlantic region who were at least “a little familiar” 
with the REL program, 42 percent reported that they attended a live or virtual event in 
the past 12 months; 34 percent received technical assistance from the REL; 33 percent 
received a response to a data or research request; and 32 percent obtained information 
from the REL’s website. Of district administrators in the Mid-Atlantic region who were 
at least “a little familiar” with the REL program, 23 percent attended a live or virtual 
event; 36 percent obtained information from the REL’s website; 14 percent received a 
response from a data or research request; and 9 percent received technical assistance 
(Figure 6-2). 

Figure 6-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Mid-Atlantic: School year 2011-12 



Used one or more services 


A live or virtual event 


Technical assistance 


Responses to data or research requests 


Information on the REL's website 


I 





42 



J 23 



1 34 

flT ^ 



33 

ri4 



32 

36 


47 

48 


0 20 40 


60 


□ State 

□ District 


80 100% 


NOTE: The total N for state administrators was 22; the total N for district administrators was 226. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


33 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Mid- Atlantic region. 
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■ Of administrators in the Mid-Atlantic region who were at least “a little familiar” with the 
REL program 41 percent of state administrators and 34 percent of district 
administrators did not use any REL services in the past 12 months. ,4 When asked why 
they had not used any REL services in the past year, the most common responses for 
state administrators were that their needs were met elsewhere (43%) or they didn’t know 
what services were available (41%). Similarly, the most common responses for district 
administrators (60 and 55 percent, respectively) were that they didn’t know what 
services were available or their needs were met elsewhere (Table 6-7). 



Table 6-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Mid-Atlantic: School year 2011-12 


Reason 

State 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

4 

43 

41 

55 

Didn’t know what services were available 

4 

41 

44 

60 

Had no need for REL resources 

3 

37 

11 

15 

Not a good match between their current needs and the 
REL’s resources 

X 

X 

5 

7 

REL that served their state did not have a good 
reputation 

0 

0 

X 

X 


NOTE: The total N for state administrators was 9, and the total N for district administrators was 74. % Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Mid-Atlantic region who had used REL services were also asked about the types of contact 
they had in the past 1 2 months with the REL. 

■ Majorities of state administrators in the Mid-Atlantic region who were at least “a little 
familiar” with the REL program and had used REL Mid-Atlantic services in the past 
year indicated that they attended a meeting or workshop at which a REL representative 
was present (91%); contacted the REL for research or other assistance (72%); and 
attended a REL-sponsored conference, training, or workshop (61%). Among district 
administrators in the Mid-Atlantic region who were at least “a little familiar” with the 
REL program and had used REL Mid-Atlantic services in the past year, 40 percent said 
they attended a meeting at which a REL representative was present; 37 percent said they 
attended a REL-sponsored conference, training, or workshop; 28 percent contacted a 
reference desk for help or used the “Ask a REL” link on the REL’s website; 25 percent 
said that they or their organization contacted the REL for research or other assistance; 
and 14 percent said they forwarded someone else’s request to the REL (Table 6-8). 


34 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 6-8. Percentage of administrators who had used REL services and reported having 
various types of contact with the REL serving their state in the past year— Mid- 
Atlantic: School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

3 

29 

27 

28 

Attended a REL-sponsored conference, training, or 
workshop 

6 

61 

34 

37 

A REL representative was present at a meeting or 
workshop 

9 

91 

38 

40 

Contacted REL for research or other assistance 

7 

72 

24 

25 

Forwarded a request to the REL 

t 

X 

13 

14 

Other type of contact 

0 

0 

10 

11 


NOTE: The total N for state administrators was 10, and the total N for district administrators was 106. X Reporting standards were not 
met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 


Twenty-eight percent of state administrators and 26 percent of district administrators in the 
Mid-Atlantic region who were at least “a. little familiar” with the REL program overall were 
“very satisfied” with it. 


■ Of the district administrators in the Mid-Atlantic region who were at least “a little 
familiar” with the REL program overall, 60 percent reported being “somewhat 
satisfied” with it and 14 percent reported being “not at all satisfied” with it (Table 6-9). 

Table 6-9. Percentage of the region’s administrators at least “a little familiar” with the REL 
program who were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” 
with it— Mid-Atlantic: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

4 

28 

47 

26 

Somewhat satisfied 

X 

X 

105 

60 

Not at all satisfied 

X 

X 

25 

14 


NOTE: Percentages may not sum to 100 due to rounding. | Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Regional Educational Laboratory Midwest serves the following states: 

■ I ll inois; 

■ Indiana; 

■ Iowa; 

■ Michigan; 

■ Minnesota; 

■ Ohio; and 

■ Wisconsin. 

For the 2006-1 1 contract period, REL Midwest was headquartered at Learning Point Associates 
(LPA), located in Naperville, Illinois. LPA is an affiliate of the American Institutes for Research 
(AIR). At the time of data collection, LPA had held previous REL contracts. The REL Midwest 
contract beginning in FY 2012 was awarded to AIR. 


What were the technical quality and relevance of REL Midwest impact study 
reports published by iES and of the corresponding proposals ? 35 


As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 


35 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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Between March 1, 2006, and September 1, 2011, IES did not publish any impact studies from REL 
Midwest, but one proposal was reviewed as part of the evaluation: 

■ Improving Adolescent Literacy Across the Curriculum in High Schools (Content Literacy Continuum, 
CLC). 

The Content Literacy Continuum (CLC) combines whole-school and targeted approaches to 
supporting student literacy and content learning, using instructional routines and learning strategies. 
This study examined the impacts of the CLC on high school students’ reading comprehension and 
accumulation of credits in core subject areas. The study found no statistically significant impacts of 
CLC on reading comprehension or accumulation of core credits. 

The average quality rating for the impact study proposal from REL Midwest that was 
reviewed by the expert panel was 2.89. On a 5-point scale (1-5, with 1 being the lowest rating 
and 5 the highest), the average proposal quality rating fell between “weak” and “adequate” 
(Table 7-1). 

Table 7-1. Expert panel quality and relevance ratings for impact study proposals from REL 
Midwest (on a 5-point scale with 5 being the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

2.89 

3.47 


Table Reads: For the proposal Improving Adolescent Literacy Across the Curriculum in High Schools (Content Literacy Continuum, CLC), 
the mean quality dimension rating was 2.89. 


NOTE: The mean quality rating for proposals for REL Midwest was based on 27 indicator-specific ratings, and the mean relevance rating 
for proposals for REL Midwest was based on 15 indicator-specific ratings. SOURCES: Rubric and Scoring Form for the Quality and 
Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals) (Appendix A). 

The average relevance rating for the impact study proposal from REL Midwest that was 
reviewed by the expert panel was 3.47. On a 5-point scale (1-5, with 1 being the lowest rating 
and 5 the highest), the average proposal relevance rating fell between “adequate” and 
“relevant” (Table 7-1). 

Table 7-2 displays, for each indicator of quality and relevance, the mean rating from the expert panel 
review of the proposal from REL Midwest. 
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REL: Midwest 

Mean ratings from expert panel review of impact studies for REL Midwest, by rating 
indicator 


Indicators for proposals 

Proposals 

(/v=D 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

2.67 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

2.33 

ID. The design for random assignment is rigorous. 

3.67 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.00 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

2.33 

1G. The data collection plan is appropriate for the research questions. 

2.67 

1H. The data analyses will use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

3.33 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

4.00 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.00 

2B. The proposal provides a thorough summary of key literature and/or previous research in 
the topic area. 

3.00 

2C. The proposal provides a strong justification for selecting the particular intervention that 
is being studied. 

3.33 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

3.67 

2E. The proposal is clearly written and well presented. 

3.33 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


How relevant and useful were the REL Midwest technical assistance products to 
the needs of the states and districts in the region? 


Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Midwest technical assistance products were in 
meeting the needs of administrators in the Midwest region. State and district administrators were 
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included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Midwest. 36 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Midwest region was college or career readiness 
(60%). The most commonly reported area of “high need” for education research and/or 
technical assistance among district administrators in the Midwest region was other type of 
education research and/or technical assistance (39%). 

Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the six largest 
percentages of state administrators in the Midwest region indicated “high need” were: 

■ College or career readiness (60%); 

■ Using data for decisions (57%); 

■ Teacher/ staff evaluation (55%); 


36 Results for the nation are presented in Chapter 3. 
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■ Achievement gaps (54%); 

■ School accountability (48%); and 

■ Support for low achieving schools (48%). 

In the Midwest region, the five topic areas with the largest percentage of district administrators 
reporting “high need” were: 

■ Other type of education research and/ or technical assistance (39%); 

■ Using data for decisions (39%); 

■ Assessment (36%); 

■ Content standards, curriculum, or instmction in areas other than reading/ writing, or 
STEM (36%); and 

■ Content standards, curriculum or instmction in reading/writing (33%). 

Detail on the need for other areas of research and technical assistance is provided in Table 7-3. 



Eighteen percent of state administrators and 27 percent of district administrators in the 
Midwest region reported that their education research and technical assistance needs were 
met “very well” (as opposed to “moderately well” or “not well”), taking into account all 
sources of such research and technical assistance. 

■ Seventy-one percent of state administrators in the Midwest region reported that their 
education research and technical assistance needs were met “moderately well” and 1 1 
percent reported that their needs were “not well” met by their sources of assistance 
(Table 7-4). 

■ Sixty-five percent of district administrators in the Midwest region reported that their 
education research and technical assistance needs were met “moderately well” and 8 
percent reported that their needs were “not well” met by their sources of assistance 
(Table 7-4). 
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Table 7-3. 


Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Midwest: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

43 

54 

37 

9 

425 

32 

50 

18 

Assessment (formative or summative) 

43 

46 

45 

9 

425 

36 

47 

17 

Behavior, character education, or health 

41 

10 

41 

49 

425 

17 

44 

39 

College or career readiness 

42 

60 

30 

10 

424 

29 

42 

28 

Content standards, curriculum or instruction in STEM 

42 

41 

43 

16 

425 

36 

47 

17 

Content standards, curriculum or instruction in reading/writing 

42 

42 

40 

18 

424 

33 

47 

20 

Content standards, curriculum or instruction in other areas 

42 

9 

58 

34 

422 

10 

58 

33 

Dropout prevention 

42 

34 

49 

17 

424 

20 

34 

46 

Early childhood 

42 

41 

43 

16 

424 

21 

38 

41 

English language learners 

42 

45 

48 

7 

425 

19 

36 

45 

High school reform 

42 

35 

49 

16 

425 

19 

37 

44 

Leadership 

42 

35 

42 

24 

424 

19 

56 

25 

Longitudinal data systems 

41 

44 

28 

29 

425 

28 

44 

28 

Parental involvement 

41 

28 

45 

28 

423 

25 

48 

27 

Professional development 

42 

33 

51 

17 

424 

27 

52 

21 

Rural schools 

42 

24 

52 

24 

423 

16 

31 

53 

School accountability 

43 

48 

27 

25 

424 

18 

48 

33 

School choice 

42 

17 

31 

52 

423 

6 

23 

71 

School finance 

42 

20 

30 

50 

424 

19 

38 

42 

Students with disabilities 

42 

30 

55 

16 

425 

25 

48 

26 

Supplemental education services 

42 

* 

t 

56 

423 

12 

42 

47 

Support for low-achieving schools 

42 

48 

36 

17 

425 

23 

32 

45 

Teacher/staff evaluation 

42 

55 

33 

12 

424 

32 

44 

24 

Using data for decisions 

43 

57 

30 

13 

423 

39 

42 

19 

Other 

0 

0 

0 

0 

44 

39 

11 

50 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. X Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Table 7-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well," or “not well," taking into 
account all sources of assistance— Midwest: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

8 

18 

116 

27 

Moderately well 

30 

71 

277 

65 

Not well 

5 

11 

32 

8 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported source of education research and/or technical assistance for state 
administrators was professional associations (86%), and for district administrators was 
counterparts in other states and districts (82%). Fifty- five percent of state administrators and 
15 percent of district administrators in the Midwest region reported that they relied on the 
REL program “to a great extent” or “to a moderate extent” for research and/or technical 
assistance. 


■ State and district administrators in the Midwest region reported that they used a variety 
of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on professional associations (86%) and 
counterparts in other states and districts (76%), while district administrators were most 
likely to rely “to a great extent” or “to a moderate extent” on counterparts in other 
states and districts (82%) and journals and publications (75%) (Figure 7-1). 

■ Fifty-five percent of state administrators and 1 5 percent of district administrators in the 
Midwest region reported that they relied on the REL program nationwide “to a great 
extent” or “to a moderate extent” for research and/or technical assistance (Figure 7-1). 

■ Twenty-six percent of state administrators in the Midwest region reported that it was 
“very easy” (as opposed to “moderately easy” or “not at all easy”)to access education 
research and/or technical assistance across the available sources of information, and 40 
percent of district administrators in the Midwest region reported that it was “very easy” 
to access such assistance easy (Table 7-5). 
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Figure 7-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Midwest: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from 
ED, including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



0 20 40 60 80 100 % 


NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works." No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on “other sources” of education research and/or technical 
assistance. The total Ns for state and district administrators on the item about reliance on the REL program were 43 and 425, 
respectively. The total N for state administrators on the items about other specified sources of research ranged from 42 to 43. The 
total N for district administrators on the items about other specified sources of research ranged from 422 to 425, depending on the 
number of district respondents who chose not to respond to an individual item; and the total N for district administrators for “other 
sources” was 24. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Table 7-5. Percentage of all administrators who reported that it was ’’very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Midwest: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

11 

26 

168 

40 

Moderately easy to access 

24 

56 

223 

52 

Not at all easy to access 

8 

18 

34 

8 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How familiar were state and district administrators with the REL program? 


Eighty-seven percent of state administrators in the Midwest region reported being at least 
“a little familiar” with the REL program, compared with 49 percent of district 
administrators. 


■ Fifty-two percent of district administrators in the Midwest region reported that they 
were "not familiar at all” with the REL program, compared to 13 percent of state 
administrators (Table 7-6)” 

Table 7-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all” with the REL program 
overall —Midwest: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

12 

27 

11 

3 

Somewhat familiar 

15 

37 

66 

16 

A little familiar 

10 

23 

126 

30 

Not familiar at all 

6 

13 

222 

52 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How many state and district administrators used REL services? 


Eighty-three percent of state administrators and 41 percent of district administrators in the 
Midwest region who were atleast “ a little familiar” with the REL program reported that 
they used one or more REL services in the past 12 months. 37 


37 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Midwest region. 
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Figure 7-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Midwest: School year 2011-12 


Used one of more services 


A live or virtual event 


Technical assistance 


Responses to data or research requests 


Information on the REL's website 


13 


11 


20 


41 


46 


51 


38 


30 


63 


40 


60 


83 


□ State 

□ District 


80 


100% 


NOTE: The total N for state administrators was 37; the total N for district administrators was 203. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Majorities of state administrators in the Midwest region who were at least “a little 
familiar” with the REL program reported that they used each of two types of REL 
services in the past 12 months: technical assistance (63%) and responses to data or 
research requests (51%). Of district administrators in the Midwest region who were at 
least “a little familiar” with the REL program, 30 percent obtained information from the 
REL’s website; 13 percent attended a live or virtual event; 1 1 percent received a 
response from a data or research request; and 7 percent received technical assistance 
(Figure 7-2). 

■ Of administrators in the Midwest region who were at least “a little familiar” with the 
REL program 34 percent of district administrators did not use any REL services in the 
past 12 months. ,h When asked why they had not used any REL services in the past year, 
the most common responses (59 and 59 percent, respectively) were that their needs 
were met elsewhere or they didn’t know what services were available (Table 7-7). 


38 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 7-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Midwest: School year 2011-12 


Reason 

State administrators 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

t 

t 

41 

59 

Didn’t know what services were available 

t 

t 

41 

59 

Had no need for REL resources 

t 

t 

10 

14 

Not a good match between their current needs and the 
REL’s resources 

t 

t 

4 

6 

REL that served their state did not have a good 
reputation 

0 

0 

0 

0 


NOTE: The total N for state administrators was 5, and the total N for district administrators was 69. t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Midwest region who had used REL services were also asked about the types of contact they 
had in the past 1 2 months with the REL. 


■ Majorities of state administrators in the Midwest region who were at least “a little 

familiar” with the REL program and had used REL Midwest services in the past year 
indicated that they or their organization contacted the REL for research or other 
assistance (90%), or they attended a meeting or workshop at which a REL 
representative was present (78%). Among district administrators in the Midwest region 
who were at least “a little familiar” with the REL program and had used REL Midwest 
services in the past year, 33 percent said they attended a meeting at which a REL 
representative was present; 29 percent contacted a reference desk for help or used the 
“Ask a REL” link on the REL’s website; 27 percent said they attended a REL- 
sponsored conference, training, or workshop; and 25 percent said that they or their 
organization contacted the REL for research or other assistance 
(Table 7-8). 
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Table 7-8. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Midwest: 
School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

t 

t 

21 

29 

Attended a REL-sponsored conference, training, or 
workshop 

14 

48 

20 

27 

A REL representative was present at a meeting or 
workshop 

23 

78 

24 

33 

Contacted REL for research or other assistance 

26 

90 

18 

25 

Forwarded a request to the REL 

6 

20 

11 

15 

Other type of contact 

t 

t 

8 

11 


NOTE: The total N for state administrators was 30, and the total N for district administrators was 78. X Reporting standards were not 
met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 

Fifty-six percent of state administrators and 17 percent of district administrators in the 
Midwest region who were atleast “ a little familiar” with the REL program overall were 
“very satisfied” with it. 

■ Of the state administrators in the Midwest region who were at least “a little familiar” 
with the REL program overall, 44 percent reported being “somewhat satisfied” with it 
and 0 percent of state administrators reported being “not at all satisfied” with it (Table 
7-9). 

■ Of the district administrators in the Midwest region who were at least “a little familiar” 
with the REL program overall, 62 percent reported being “somewhat satisfied” with it 
and 21 percent reported being “not at all satisfied” with it (Table 7-9). 
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Table 7-9. Percentage of the region’s administrators at least “a little familiar” with the REL 

program who were “very satisfied, “somewhat satisfied,” or “not at all satisfied” with 
it— Midwest: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

19 

56 

27 

17 

Somewhat satisfied 

15 

44 

101 

62 

Not at all satisfied 

0 

0 

34 

21 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Regional Educational Laboratory Northeast and Islands serves the following jurisdictions: 

■ Connecticut; 

■ Maine; 

■ Massachusetts; 

■ New Hampshire; 

■ New York; 

■ Puerto Rico; 

■ Rhode Island; 

■ U.S. Virgin Islands; and 

■ Vermont. 

For the 2006-1 1 contract period, REL Northeast and Islands was based at the Education 
Development Center (EDC) in Newton, Massachusetts. Although EDC held one of the original 
REL contracts, it did not hold the previous REL contract. EDC was also awarded the REL 
Northeast and Islands contract beginning in FY 2012. 
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What were the technical quality and relevance of REL Northeast and Islands 
impact study reports published by IES and of the corresponding proposals ? 39 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 

Between March 1, 2006, and September 1, 201 1, IES published one impact study from REL 
Northeast and Islands: 

■ Impact of the Thinking Reader® Sofhvare Program on Grade 6 Reading Vocabnlaiy, 

Comprehension, Strategies, and Motivation: Pinal Report. 

Thinking Reader® is a software program that aims to motivate middle school students to read and to 
make self-directed use of seven target comprehension strategies: (a) summarizing, (b) clarifying, (c) 
visualizing, (d) reflecting, (e) questioning, (f) predicting, and (g) feeling. Students listen to a novel 
while following highlighted text on a computer screen and then respond to questions about the 
story. The study assessed the effectiveness of Thinking Reader® by comparing the reading 
comprehension of students in the Thinking Reader® and comparison conditions at the end of the 
school year. The study found no statistically significant differences on the comprehension outcomes 
of students in the Thinking Reader® classes, compared with students in the comparison classes. 

The average quality ratings for the impact study report and proposal from REL Northeast 
and Islands that were reviewed by the expert panel were 4.18 and 3.63, respectively. On a 5- 
point scale (1-5, with 1 being the lowest rating and 5 the highest), the average report quality 
rating fell between “strong” and “very strong,” and the average proposal quality rating fell 
between “adequate” and “strong” (Table 8-1). 


39 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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Table 8-1. Expert panel quality and relevance ratings for lES-published impact study reports 
and corresponding proposals from REL Northeast and Islands (on a 5-point scale 
with 5 being the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

3.63 

3.60 

lES-published impact study report 

4.18 

3.67 


Table Reads: For the proposal Impact of the Thinking Reader® Software Program on Grade 6 Reading Vocabulary, Comprehension, 
Strategies, and Motivation, the mean quality dimension rating was 3.63. 


NOTE: The mean quality rating for proposals for REL Northeast and Islands was based on 27 indicator-specific ratings, and the mean 
relevance rating for proposals for REL Northeast and Islands was based on 15 indicator-specific ratings. The mean quality rating for 
reports for REL Northeast and Islands was based on 45 indicator-specific ratings, and the mean relevance rating for reports for REL 
Northeast and Islands was based on 18 indicator-specific ratings. SOURCES: Rubric and Scoring Form for the Quality and Relevance of 
Regional Educational Laboratories’ Impact Study Projects (for reports); Rubric and Scoring Form for the Quality and Relevance of 
Regional Educational Laboratories’ Impact Study Projects (for proposals) (Appendix A). 


The average relevance ratings for the impact study report and proposal from REL Northeast 
and Islands that were reviewed by the expert panel were 3.67 and 3.60, respectively. On a 5- 
point scale (1-5, with 1 being the lowest rating and 5 the highest), the average report and 
proposal relevance ratings fell between “adequate” and “relevant” (Table 8-1). 

Table 8-2 displays, for each indicator of quality and relevance, the mean ratings from expert panel 
reviews of the IES-published impact study and corresponding proposal from REL Northeast and 
Islands. 
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Table 8-2. Mean ratings from expert panel review of impact studies for REL Northeast, by 
rating indicator 


Indicators for proposals 
Quality 

Proposals 

(/v=D 

1A. The intervention for the treatment group and the condition for the control group 
are clearly described. 

4.00 

IB. The plan to document implementation of the intervention (e.g., exposure, quality 
of delivery, adherence) is appropriate. 

3.33 

1C. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.33 

ID. The design for random assignment is rigorous. 

4.00 

IE. The sampling strategies are appropriate for targeted populations, and the 

resulting sample size(s) for the impact questions will have adequate statistical 
power. 

3.67 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.33 

1G. The data collection plan is appropriate for the research questions. 

3.33 

1H. The data analyses will use appropriate methods to address the research questions 
(e.g., adjusting where necessary for any clustering, missing data from 
nonresponse, multiple comparisons, and/or baseline non-equivalence between 
groups). 

3.00 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

Relevance 

3.67 

2A. The proposal provides a strong justification why the topic was selected as salient 
to meet the needs of the region served by the REL. 

3.67 

2B. The proposal provides a thorough summary of key literature and/or previous 
research in the topic area. 

3.33 

2C. The proposal provides a strong justification for selecting the particular intervention 
that is being studied. 

3.33 

2D. The report will contribute new information about the effectiveness of the 
intervention being studied and the more general topic being addressed. 

4.00 

2E. The proposal is clearly written and well presented. 

3.67 
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Table 8-2. Mean ratings from expert panel review of impact studies for REL Northeast, by 
rating indicator (continued) 


Indicators for reports 

lES-published 

reports 

(/V=l) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

4.00 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

4.00 

1C. There is minimal contamination in the form of crossover between subjects in 

treatment and control condition or spillover of the intervention from the treatment 
to the control group. 

4.33 

ID. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.67 

IE. The design and implementation of random assignment are rigorous. 

3.33 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

4.00 

1G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 

4.00 

1H. The data collection plan is appropriate for the research questions. 

4.00 

11. The data collection plan is well implemented. 

4.67 

U. The overall attrition rate and differential attrition rates are acceptable given the 
length of the intervention. 

4.67 

IK. The data analyses use appropriate methods to address the research questions 

(e.g., adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

4.67 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.00 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

4.33 

IN. All of the research questions are specifically addressed by the analyses. 

5.00 

10. The limitations of the study are clearly and comprehensively stated. 

3.00 

Relevance 

2A. The report provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

3.67 

2B. The report provides a thorough summary of key literature and/or previous research 
in the topic area. 

4.00 

2C. The report provides a strong justification for selecting the particular intervention that 
is being studied. 

3.33 

2D. The report contributes new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.00 

2E. The executive summary of the report is easy to read and understand for a lay 
audience. 

3.00 

2F. The report is clear and well written for the technical audience. 

4.00 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


Source: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports); 
Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals) 
(Appendix A). 
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How relevant and useful were the REL Northeast and Islands technical 
assistance products to the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Northeast and Islands technical assistance 
products were in meeting the needs of administrators in the Northeast and Islands region. State and 
district administrators were included in the sample regardless of previous use of REL services or 
familiarity with the REL program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Northeast and Islands. 40 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Northeast and Islands region was English 
language learners (49%). The most commonly reported area of “high need” for education 
research and/or technical assistance among district administrators in the Northeast and 
Islands region was using data for decisions (36%). 


40 Results for the nation are presented in Chapter 3. 
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Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need for research/ assistance in specific topic areas. The areas in which the five largest 
percentages of state administrators in the Northeast and Islands region indicated “high need” were: 

■ English language learners (49%); 

■ Using data for decisions (45%); 

■ Achievement gaps (45%); 

■ Dropout prevention (43%); and 

■ Teacher/ staff evaluation (41%). 

In the Northeast and Islands region, the six topic areas with the largest percentage of district 
administrators reporting “high need” were: 

■ Using data for decisions (36%); 

■ Longitudinal data system (35%); 

■ Content standards, curriculum, or instmction in areas other than reading/ writing, or 
STEM (34%); 

■ Other types of education research and/ or technical assistance (34%); 

■ Content standards, curriculum, or instmction in reading/writing (33%); and 

■ Achievement gaps (33%). 

Details on the need for other areas of research and technical assistance are provided in Table 8-3. 
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Table 8-3. 


Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Northeast and Islands: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

35 

45 

36 

19 

376 

33 

42 

25 

Assessment (formative or summative) 

35 

24 

52 

24 

375 

32 

47 

20 

Behavior, character education, or health 

35 

16 

31 

53 

371 

16 

44 

40 

College or career readiness 

35 

40 

47 

13 

369 

25 

46 

29 

Content standards, curriculum or instruction in STEM 

35 

30 

47 

23 

374 

34 

45 

21 

Content standards, curriculum or instruction in reading/writing 

35 

32 

37 

31 

374 

33 

48 

20 

Content standards, curriculum or instruction in other areas 

35 

12 

49 

39 

371 

13 

51 

37 

Dropout prevention 

35 

43 

31 

26 

371 

20 

40 

41 

Early childhood 

35 

24 

44 

32 

373 

17 

43 

40 

English language learners 

35 

49 

25 

27 

374 

18 

39 

43 

High school reform 

35 

31 

31 

38 

374 

24 

36 

40 

Leadership 

35 

37 

33 

30 

372 

23 

47 

30 

Longitudinal data systems 

35 

40 

29 

32 

373 

35 

44 

21 

Parental involvement 

34 

24 

61 

15 

373 

20 

48 

32 

Professional development 

35 

40 

42 

19 

374 

24 

49 

27 

Rural schools 

35 

18 

29 

54 

374 

12 

21 

66 

School accountability 

35 

31 

30 

39 

374 

13 

47 

40 

School choice 

35 

* 

t 

63 

372 

4 

22 

74 

School finance 

35 

12 

21 

67 

373 

14 

36 

50 

Students with disabilities 

35 

34 

40 

27 

374 

24 

50 

26 

Supplemental education services 

35 

11 

18 

71 

374 

9 

41 

50 

Support for low-achieving schools 

35 

38 

44 

18 

373 

18 

38 

44 

Teacher/staff evaluation 

35 

41 

43 

16 

375 

32 

46 

22 

Using data for decisions 

34 

45 

36 

18 

375 

36 

46 

18 

Other 

3 

0 

0 

100 

39 

34 

13 

53 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. X Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Thirty-five percent of state administrators and 20 percent of district administrators in the 
Northeast and Islands region reported that their education research and technical assistance 
needs were met “very well” (as opposed to “moderately well” or “note well”), taking into 
account all sources of such research and technical assistance. 


■ Fifty-six percent of state administrators in the Northeast and Islands region reported 

that their education research and technical assistance needs were met “moderately well,” 
and 9 percent reported that their needs were “not well” met by their sources of 
assistance(Table 8-4). 

Table 8-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Northeast and Islands: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

12 

35 

77 

20 

Moderately well 

19 

56 

267 

71 

Not well 

3 

9 

32 

9 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Seventy-one percent of district administrators in the Northeast and Islands region 
reported that their education research and technical assistance needs were met 
“moderately well,” and 9 percent reported that their needs were “not well” met by their 
sources of assistance (Table 8-4). 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported sources of education research and/or technical assistance for both state 
and district administrators were counterparts in other states and districts (86 and 85 percent, 
respectively) and professional associations (86 percent and 81 percent, respectively). Forty- 
eight percent of state administrators and 16 percent of district administrators in the 
Northeast and Islands region reported that they relied on the REL program “to a great 
extent” or “to a moderate extent” for research and/ or technical assistance. 

■ State and district administrators in the Northeast and Islands region reported that they 
used a variety of sources for meeting their research and/ or technical assistance needs. 
State and district administrators were most likely to rely “to a great extent” or “to a 
moderate extent” (as opposed to a “small extent” or not at all) on counterparts in other 
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states and districts (86 and 85 percent, respectively) and professional associations (86 
and 81 percent, respectively). (Figure 8-1). 

Figure 8-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Northeast & Islands: School year 2011-2012 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from 
ED, including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on "other sources” of education research and/or technical 
assistance. The total Ns for state and district administrators on the item about reliance on the REL program were 35 and 376, 
respectively. The total N for state and district administrators on the items about other specified sources of research ranged from 33 to 
34 for state administrators and 372 to 376 for district administrators, depending on the number of respondents who chose not to 
respond to an individual item. The total N for district administrators for “other sources” was 22. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Forty-eight percent of state administrators and 16 percent of district administrators in 
the Northeast and Islands region reported that they relied on the REL program 
nationwide “to a great extent” or “to a moderate extent” for research and/ or technical 
assistance (Figure 8-1). 

■ Forty percent of state administrators in the Northeast and Islands region reported that it 
was “very easy” (as opposed to “moderately easy” or “not at all easy”) to access 
education research and/ or technical assistance across the available sources of 
information, and 37 percent of district administrators in the Northeast and Islands 
region reported that it was “very easy” to access such assistance (Table 8-5). 
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Table 8-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Northeast and Islands: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

14 

40 

140 

37 

Moderately easy to access 

15 

45 

200 

53 

Not at all easy to access 

5 

15 

37 

10 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Eighty-one percent of state administrators in the Northeast and Islands region reported 
being at least “ a little familiar” with the REL program, compared with 50 percent of district 
administrators. 


■ Fifty percent of district administrators in the Northeast and Islands region reported that 
they were “not familiar at all” with the REL program compared to 19 percent of state 
administrators (Table 8-6). 

Table 8-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all” with the REL program 
overall— Northeast and Islands: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

15 

43 

22 

6 

Somewhat familiar 

9 

25 

69 

18 

A little familiar 

5 

13 

99 

26 

Not familiar at all 

6 

19 

187 

50 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How many state and district administrators used REL services? 


Ninety-three percent of state administrators and 44 percent of district administrators in the 
Northeast and Islands region who were at least “ a little familiar” with the REL program 
reported that they used one or more REL services in the past 12 months. 41 

Figure 8-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Northeast and Islands: School year 2011-12 


Used one or more services 


A live or virtual event 


Technical assistance 


Responses to data or research 
requests 


Information on the REL's website 


15 




20 


44 


40 


59 



- 

13 



37 


55 


57 


60 


67 


93 


EState 
□ District 


80 


100 % 


NOTE: The total N for state administrators was 29; the total N for district administrators was 190. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Majorities of state administrators in the Northeast and Islands region who were at least 
“a little familiar” with the REL program reported that they used each of four types of 
REL services in the past 12 months: technical assistance (67%), a live or virtual event 
(59%), information on the REL’s website (57%), and responses to data or research 
requests (55%). Of district administrators in the Northeast and Islands region who were 
at least “a little familiar” with the REL program, 37 percent obtained information from 
the REL’s website; 1 5 percent attended a live or virtual event; 1 3 percent received a 


41 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Northeast and Islands region. 
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response from a data or research request; and 6 percent received technical assistance 
(Figure 8-2). 

■ Of administrators in the Northeast and Islands region who were at least “a little familiar” 
with the REL program, 42 percent of district administrators did not use any REL 
services in the past 12 months. 4 " When asked why they had not used any REL services 
in the past year, the most common responses for district administrators (64 and 50 
percent, respectively) were that they didn’t know what services were available or their 
needs were met elsewhere (Table 8-7). 

Table 8-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Northeast and Islands: School year 
2011-12 


Reason 

State administrators 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

t 

t 

40 

50 

Didn’t know what services were available 

t 

t 

51 

64 

Had no need for REL resources 

t 

t 

12 

15 

Not a good match between their current needs and the 
REL’s resources 

t 

t 

7 

9 

REL that served their state did not have a good 
reputation 

t 

t 

t 

t 


NOTE: The total N for district administrators was 80. $ Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Northeast and Islands region who had used REL services were also asked about the types of 
contact they had in the past 12 months with the REL. 

■ Majorities of state administrators in the Northeast and Islands region who were at least 
“a little familiar” with the REL program and had used REL Northeast and Islands 
services in the past year indicated that they attended a REL-sponsored conference, 
training, or workshop (81%); they or their organization contacted the REL for research 
or other assistance (80%); or they attended a meeting or workshop at which a REL 
representative was present (73%). Among district administrators in the Northeast and 
Islands region who were at least “a little familiar” with the REL program and had used 
REL Northeast and Islands services in the past year, 47 percent contacted a reference 
desk for help or used the “Ask a REL” link on the REL’s website; 30 percent said they 
attended a meeting at which a REL representative was present; 29 percent said that they 
or their organization contacted the REL for research or other assistance; and 23 percent 
said they attended a REL-sponsored conference, training, or workshop (Table 8-8). 


42 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 8-8. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Northeast 
and Islands: School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

12 

45 

33 

47 

Attended a REL-sponsored conference, training, or 
workshop 

21 

81 

16 

23 

A REL representative was present at a meeting or 
workshop 

19 

73 

21 

30 

Contacted REL for research or other assistance 

21 

80 

20 

29 

Forwarded a request to the REL 

11 

40 

5 

7 

Other type of contact 

X 

X 

13 

19 


NOTE: The total N for state administrators was 26, and the total N for district administrators was 78. t Reporting standards were not 
met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 


Sixty-three percent of state administrators and 28 percent of district administrators in the 
Northeast and Islands region who were at least “a. little familiar” with the REL program 
overall were “very satisfied” with it. 


■ Of the district administrators in the Northeast and Islands region who were at least “ a 
little familiar” with the REL program overall, 53 percent reported being “somewhat 
satisfied” with it and 19 percent reported being “not at all satisfied” with it (Table 8-9). 

Table 8-9. Percentage of the region’s administrators at least “a little familiar” with the REL 
program who were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” 
with it— Northeast and Islands: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

18 

63 

40 

28 

Somewhat satisfied 

X 

X 

77 

53 

Not at all satisfied 

X 

X 

28 

19 


NOTE: t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Regional Educational Laboratory Northwest serves the following states: 

■ Alaska; 

■ Idaho; 

■ Montana; 

■ Oregon; and 

■ Washington. 

For the 2006-1 1 contract period, REL Northwest was headquartered at Education Northwest 41 in 
Portland, Oregon, which had served as a REL since 1966; it was also awarded the REL Northwest 
contract beginning in FY 2012. 


What were the technical quality and relevance of REL Northwest impact study 
reports published by IES and of the corresponding proposals ? 44 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 2011, IES published one impact study from REL 
Northwest: 


43 The organization was previously called the Northwest Regional Educational Laboratory. 

44 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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■ An Experimental Study of the Project CRISS Reading Program on Grade 9 Reading Achievement in 
Rural High Schools. 

Through Project CRISS (Creating Independence Through Student-owned Strategies ), high school teachers learn 
how to apply research-based learning principles and reading/writing strategies in all major subject or 
content areas using materials, training, and follow-up support provided by the developer. The study 
examined the effectiveness of Project CRISS on grade 9 student reading achievement. The study did 
not find any statistical significant difference in student reading comprehension test scores between 
treatment and control conditions. 


The average quality ratings for the impact study report and proposal from REL Northwest 
that were reviewed by the expert panel were 3.64 and 3.00, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report and proposal 
quality ratings fell between “adequate” and “strong” (Table 9-1). 

Table 9-1. Expert panel quality and relevance ratings for lES-published impact study reports 
and corresponding proposals from REL Northwest (on a 5-point scale with 5 being 
the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

3.00 

3.47 

lES-published impact study report 

3.64 

3.67 


Table Reads: For the proposal An Experimental Study of the Project CRISS Reading Program on Grade 9 Reading Achievement in Rural 
High Schools, the mean quality dimension rating was 3.00. 


NOTE: The mean quality rating for proposals for REL Northwest was based on 27 indicator-specific ratings, and the mean relevance 
rating for proposals for REL Northwest was based on 15 indicator-specific ratings. The mean quality rating for reports for REL 
Northwest was based on 45 indicator-specific ratings, and the mean relevance rating for reports for REL Northwest was based on 18 
indicator-specific ratings. SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ 
Impact Study Projects (for reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories' 
Impact Study Projects (for proposals) (Appendix A). 


The average relevance ratings for the impact study report and proposal from REL 
Northwest that were reviewed by the expert panel were 3.67 and 3.47, respectively. On a 5- 
point scale (1-5, with 1 being the lowest rating and 5 the highest), the average report and 
proposal relevance ratings fell between “adequate” and “relevant” (Table 9-1). 
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Table 9-2 displays, for each indicator of quality and relevance, the mean rating from the expert panel 
review of the IES-published impact study and corresponding proposal from REL Northwest. 

Table 9-2. Mean ratings from expert panel review of impact studies for REL Northwest, by 
rating indicator 


Indicators for proposals 

Proposals 

(/V=l) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

2.67 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

3.00 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

2.67 

ID. The design for random assignment is rigorous. 

3.00 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.00 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.33 

1G. The data collection plan is appropriate for the research questions. 

2.67 

1H. The data analyses will use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

3.00 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

3.67 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.00 

2B. The proposal provides a thorough summary of key literature and/or previous research in 
the topic area. 

3.00 

2C. The proposal provides a strong justification for selecting the particular intervention that 
is being studied. 

3.00 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.33 

2E. The proposal is clearly written and well presented. 

3.00 
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Table 9-2. Mean ratings from expert panel review of impact studies for REL Northwest, by 
rating indicator (continued) 


Indicators for reports 

IES- 
published 
reports 
(N = 1) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

3.00 

1C. There is minimal contamination in the form of crossover between subjects in treatment 
and control condition or spillover of the intervention from the treatment to the control 
group. 

3.33 

ID. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

4.33 

IE. The design and implementation of random assignment are rigorous. 

3.67 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

3.67 

1G. Outcome measures are valid and reliable and not overly aligned with the intervention. 

4.00 

1H. The data collection plan is appropriate for the research questions. 

3.33 

11. The data collection plan is well implemented. 

2.67 

U. The overall attrition rate and differential attrition rates are acceptable given the length 
of the intervention. 

2.33 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

4.33 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.00 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

4.67 

IN. All of the research questions are specifically addressed by the analyses. 

4.00 

10. The limitations of the study are clearly and comprehensively stated. 

4.00 

Relevance 

2A. The report provides a strong justification why the topic was selected as salient to meet 
the needs of the region served by the REL. 

4.00 

2B. The report provides a thorough summary of key literature and/or previous research in 
the topic area. 

3.67 

2C. The report provides a strong justification for selecting the particular intervention that is 
being studied. 

3.00 

2D. The report contributes new information about the effectiveness of the intervention being 
studied and the more general topic being addressed. 

4.00 

2E. The executive summary of the report is easy to read and understand for a lay audience. 

3.67 

2F. The report is clear and well written for the technical audience. 

3.67 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 
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How relevant and useful were the REL Northwest technical assistance products 
to the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Northwest technical assistance products were 
in meeting the needs of administrators in the Northwest region. State and district administrators 
were included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Northwest. 45 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Northwest region was teacher/staff evaluation 
(65%). The most commonly reported area of “high need” for education research and/or 
technical assistance among district administrators in the Northwest region was content 
standards, curriculum, or instruction in areas other than reading/writing or STEM (36%). 


45 Results for the nation are presented in Chapter 3. 
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Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the five largest 
percentages of state administrators in the Northwest region indicated “high need” were: 


■ Teacher/ staff evaluation (65%), 

■ Career or college readiness (59%), 

■ Rural school (55%), 

■ Dropout prevention (51%), and 

■ Support for low-achieving schools (50%). 


In the Northwest region, the five topic areas with the largest percentage of district administrators 
reporting “high need” were: 


■ Content standards, curriculum, or instmction in areas other than reading/ writing or 
STEM (36%); 

■ Using data for decisions (35%); 

■ Achievement gaps (33%); 

■ Professional development (31%); and 

■ Longitudinal data systems (30%). 


Details on the need for other areas of research and technical assistance are provided in 
Table 9-3. 
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Table 9-3. 


Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Northwest: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or 
no need 


High 

Moderate 

Low or 
no need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

30 

45 

33 

23 

379 

33 

45 

22 

Assessment (formative or summative) 

30 

33 

47 

21 

379 

29 

50 

21 

Behavior, character education, or health 

30 

X 

X 

55 

379 

16 

46 

37 

College or career readiness 

30 

59 

24 

16 

377 

25 

41 

33 

Content standards, curriculum or instruction in STEM 

30 

31 

39 

30 

379 

36 

43 

21 

Content standards, curriculum or instruction in reading/writing 

29 

28 

41 

31 

378 

28 

48 

24 

Content standards, curriculum or instruction in other areas 

30 

13 

37 

50 

379 

10 

46 

44 

Dropout prevention 

30 

51 

30 

19 

379 

21 

40 

39 

Early childhood 

30 

20 

37 

43 

379 

24 

41 

35 

English language learners 

30 

49 

36 

15 

379 

23 

37 

40 

High school reform 

29 

45 

18 

37 

379 

19 

36 

46 

Leadership 

30 

40 

33 

27 

378 

22 

50 

28 

Longitudinal data systems 

30 

40 

34 

26 

378 

30 

40 

30 

Parental involvement 

29 

27 

52 

21 

378 

25 

50 

25 

Professional development 

30 

32 

47 

20 

379 

31 

44 

25 

Rural schools 

30 

55 

X 

X 

378 

29 

29 

42 

School accountability 

30 

40 

33 

27 

379 

14 

47 

39 

School choice 

30 

16 

23 

61 

379 

3 

25 

73 

School finance 

30 

10 

45 

46 

378 

17 

35 

48 

Students with disabilities 

30 

19 

51 

30 

379 

27 

46 

27 

Supplemental education services 

30 

22 

15 

63 

378 

13 

37 

50 

Support for low-achieving schools 

30 

50 

37 

13 

378 

28 

39 

33 

Teacher/staff evaluation 

29 

65 

14 

21 

379 

28 

42 

30 

Using data for decisions 

30 

43 

40 

16 

379 

35 

43 

23 

Other 

30 

0 

0 

100 

31 

26 

X 

X 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. X Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Nineteen percent of state administrators and 18 percent of district administrators in the 
Northwest region reported that their education research and technical assistance needs 
were met “very well” (as opposed to “moderately well” or “not well”), taking into account 
all sources of such research and technical assistance. 


■ Seventy- two percent of district administrators in the Northwest region reported that 

their education research and technical assistance needs were met “moderately well” and 
1 1 percent reported that their needs were “not well” met by their sources of assistance 
(Table 9-4). 

Table 9-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Northwest: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

6 

19 

68 

18 

Moderately well 

t 

t 

270 

72 

Not well 

* 

t 

40 

11 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported sources of education research and/or technical assistance for state 
administrators were professional associations (84%) and counterparts in other states and 
districts (83%). The most reported sources of education research and/ or technical 
assistance for district administrators were counterparts in other states and districts (81%) 
and journals and publications (69%). Fifty-seven percent of state administrators and 22 
percent of district administrators in the Northwest region reported that they relied on the 
REL program “to a great extent” or “to a moderate extent” for research and/or technical 
assistance. 


■ State and district administrators in the Northwest region reported that they used a 
variety of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on professional associations (84%) and 
counterparts in other states and districts (83%), while district administrators were most 
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likely to rely “to a great extent” or “to a moderate” extent on counterparts in other 
states and districts (81%) and journals and publications (69%) (Figure 9-1). 

Figure 9-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Northwest: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from 
ED, including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on “other sources" of education research and/or technical 
assistance. The total N for state administrators was 30. The total N for district administrators on the item about reliance on the REL 
program was 380; the total N for district administrators on the items about other specified sources of research ranged from 373 to 
379, depending on the number of district respondents who chose not to respond to an individual item; and the total N for district 
administrators for “other sources" was 16. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Fifty-seven percent of state administrators and 22 percent of district administrators in 
the Northwest region reported that they relied on the REL program nationwide “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance (Figure 
9-1). 

■ Thirty percent of state administrators in the Northwest region reported that it was “very 
easy” (as opposed to “moderately easy” or “not at all easy”) to access education research 
and/ or technical assistance across the available sources of information, and 30 percent 
of district administrators in the Northwest region reported that it was “very easy” to 
access such assistance (Table 9-5). 
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Table 9-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Northwest: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

9 

30 

115 

30 

Moderately easy to access 

X 

X 

235 

62 

Not at all easy to access 

X 

X 

29 

8 


NOTE: t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Sixty-three percent of state administrators in the Northwest region reported being “very 
familiar” or “somewhat familiar” with the REL program, compared to 36 percent of district 
administrators. 


■ Thirty- three percent of district administrators in the Northwest region reported that 

they were “a little familiar” with the REL program, and 32 percent reported that they 
were “not familiar at all” with the REL program (Table 9-6). 

Table 9-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all" with the REL program 
overall— Northwest: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

7 

25 

29 

8 

Somewhat familiar 

12 

38 

106 

28 

A little familiar 

X 

X 

124 

33 

Not familiar at all 

X 

X 

120 

32 


NOTE: Percentages may not sum to 100 due to rounding. | Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Evaluation of the Regional Educational Laboratories: Final Report 

122 



REL: Northwest 


9 


How many state and district administrators used REL services? 

Eighty-two percent of state administrators and 55 percent of district administrators in the 
Northwest region who were atleast “ a little familiar” with the REL program reported that 
they used one or more REL services in the past 12 months. 46 

■ Majorities of state administrators in the Northwest region who were at least “a little 
familiar” with the REL program reported that they used each of two types of REL 
services in the past 12 months: a live or virtual event (59%) and technical assistance 
(52%). Of district administrators in the Northwest region who were at least “a little 
familiar” with the REL program, 42 percent obtained information from the REL’s 
website; 20 percent attended a live or virtual event; 16 percent received technical 
assistance; and 15 percent received a response from a data or research request 
(Figure 9-2). 

Figure 9-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Northwest: School year 2011-12 


Used one or more services 


A live or virtual event 


Technical assistance 


Responses to data or research requests 


Information on the REL's website 


55 




20 



52 


16 




46 


15 





45 

42 


59 


82 


IlState 
■ District 


20 


40 


60 


80 


100 % 


NOTE: The total N for state administrators was 29; the total N for district administrators was 260. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


46 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Northwest region. 
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■ Of administrators in the Northwest region who were at least “a little familiar” with the 
REL program, 1 1 percent of state administrators and 29 percent of district 
administrators did not use any REL services in the past 12 months. 4 When asked why 
they had not used any REL services in the past year, the most common response for 
state and district administrators was that their needs were met elsewhere (100 percent 
and 57 percent, respectively) (Table 9-7). 

Table 9-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Northwest: School year 2011-12 


Reason 

State administrators 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

3 

100 

41 

57 

Didn’t know what services were available 

X 

X 

36 

49 

Had no need for REL resources 

X 

X 

16 

22 

Not a good match between their current needs and the 
REL’s resources 

X 

X 

8 

11 

REL that served their state did not have a good 
reputation 

0 

0 

X 

X 


NOTE: The total N for state administrators was 3, and the total N for district administrators was 73. t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Northwest region who had used REL services were also asked about the types of contact they 
had in the past 1 2 months with the REL. 

■ Majorities of state administrators in the Northwest region who were at least “a little 

familiar” with the REL program and had used REL Northwest services in the past year 
indicated that they or their organization contacted the REL for research or other 
assistance (78%); they attended a meeting or workshop at which a REL representative 
was present (76%); and they attended a REL-sponsored conference, training, or 
workshop (61%). Among district administrators in the Northwest region who were at 
least “a little familiar” with the REL program and had used REL Northwest services in 
the past year, 37 percent said they attended a meeting at which a REL representative 
was present; 36 percent said they attended a REL sponsored conference, training, or 
workshop; 36 percent said that they or their organization contacted the REL for 
research or other assistance; and 35 percent contacted a reference desk for help or used 
the “Ask a REL” link on the REL’s website (Table 9-8). 


47 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 9-8. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Northwest: 
School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

6 

27 

42 

35 

Attended a REL-sponsored conference, training, or 
workshop 

13 

61 

43 

36 

A REL representative was present at a meeting or 
workshop 

17 

76 

45 

37 

Contacted REL for research or other assistance 

17 

78 

43 

36 

Forwarded a request to the REL 

6 

29 

14 

12 

Other type of contact 

5 

21 

25 

21 


NOTE: The total N for state administrators was 22, and the total N for district administrators was 127. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 


Thirty-nine percent of state administrators and 29 percent of district administrators in the 
Northwest region who were at least'' 1 a little familiar” with the REL program overall were 
“very satisfied” with it. 


■ Of the district administrators in the Northwest region who were at least “a little familiar” 
with the REL program overall, 56 percent reported being “somewhat satisfied” with it 
and 14 percent reported being “not at all satisfied” with it (Table 9-9). 

Table 9-9. Percentage of the region’s administrators at least “a little familiar” with the REL 
program who were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” 
with it— Northwest: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

11 

39 

68 

29 

Somewhat satisfied 

X 

X 

132 

56 

Not at all satisfied 

X 

X 

35 

14 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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HI- * 


The Pacific region also 
includes American 
Samoa, Federated 
States of Micronesia, 
Guam, Northern 
Mariana Islands, 
Republic of the 
Marshall Islands, and 
Republic of Palau, not 
pictured on the map. 


REL: Pacific 



Regional Educational Laboratory Pacific serves the following geographic locations: 


■ American Samoa; 

■ Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap); 

■ Guam; 

■ Hawaii; 

■ Northern Mariana Islands; 

■ Republic of the Marshall Islands; and 

■ Republic of Palau. 

For the 2006-1 1 contract period, REL Pacific was housed at Pacific Resources for Education and 
Learning (PREL) in Honolulu, Hawaii. PREL had held previous REL contracts. The REL Pacific 
contract beginning in FY 2012 was awarded to Mid-continent Research for Education and Learning 
(McREL). 


What were the technical quality and relevance of REL Pacific impact study 
reports published by iES and of the corresponding proposals ? 48 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 201 1, IES had not published any impact studies from 
REL Pacific, but one proposal was reviewed as part of the evaluation: 

■ Pacific Evaluation of Principles-Pased Professional Development to Improve Pea ding Comprehension 

for English Language Learners. 


48 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 


Evaluation of the Regional Educational Laboratories: Final Report 

127 



REL: Pacific 


10 


Pacific Communities with High Performance in Literacy Development (Pacific CHILD) is a two- 
year professional development program that trains fourth and fifth grade teachers in research-based 
reading comprehension strategies and instructional practices for enhancing student reading 
comprehension. The study examined the impact of Pacific CHILD on student achievement in 
reading comprehension and on teacher pedagogical knowledge and instructional practice in English 
language arts classes. The study found positive impacts of Pacific CHILD on reading 
comprehension and on teachers’ instructional practices and knowledge of theories and strategies 
related to effective reading instruction. 


The average quality rating for the impact study proposal from REL Pacific that was 
reviewed by the expert panel was 2.85. On a 5-point scale (1-5, with 1 being the lowest rating 
and 5 the highest), the average proposal quality rating fell between “weak” and “adequate” 
(Table 10-1). 

Table 10-1. Expert panel quality and relevance ratings for impact study proposals from REL 
Pacific (on a 5-point scale with 5 being the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact Study Proposal 

2.85 

3.27 


Table Reads: For the proposal Pacific Evaluation of Principles-Based Professional Development to Improve Reading Comprehension for 
English Language Learners, the mean quality dimension rating was 2.85. 


NOTE: The mean quality rating for proposals for REL Pacific was based on 27 indicator-specific ratings, and the mean relevance rating 
for proposals for REL Pacific was based on 15 indicator-specific ratings. SOURCES: Rubric and Scoring Form for the Quality and 
Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals) (Appendix A). 


The average relevance rating for the impact study proposal from REL Pacific that was 
reviewed by the expert panel was 3.27. On a 5-point scale (1-5, with 1 being the lowest rating 
and 5 the highest), the average proposal relevance rating fell between “adequate” and 
“relevant” (Table 10-1). 

Table 10-2 displays, for each indicator of quality and relevance, the mean ratings from expert panel 
review of the proposal from REL Pacific. 
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Table 10-2. Mean ratings from expert panel review of the impact study proposal for REL Pacific, 
by rating indicator 


Indicators for proposals 

Proposals 

(/v=D 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

3.00 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

3.33 

ID. The design for random assignment is rigorous. 

3.00 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

2.67 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

2.00 

1G. The data collection plan is appropriate for the research questions. 

3.00 

1H. The data analyses will use appropriate methods to address the research questions 
(e.g., adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

2.67 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

2.67 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.00 

2B. The proposal provides a thorough summary of key literature and/or previous research 
in the topic area. 

3.00 

2C. The proposal provides a strong justification for selecting the particular intervention that 
is being studied. 

3.00 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

3.33 

2E. The proposal is clearly written and well presented. 

3.00 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


Source: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports); 
Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals) 
(Appendix A). 


How relevant and useful were the REL Pacific technical assistance products to 
the needs of the states in the region? 


Between the fall of 201 1 and spring of 2012, a survey of state administrators was conducted to 
determine how relevant and useful REL Pacific technical assistance products were in meeting the 
needs of administrators in the Pacific region. State administrators were included in the sample 
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regardless of previous use of REL services or familiarity with the REL program. Specific research 
questions included: 

■ What needs did state administrators have for education research and technical 
assistance, and were those needs met? 

■ What sources of education research and technical assistance did state administrators 
use? 

■ How familiar were state administrators with the REL program? 

■ How many state administrators used REL services? 

■ How satisfied with the REL program were state administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state administrators in REL Pacific. 49 


What needs did state administrators have for education research and technical 
assistance, and were those needs met? 

The most commonly reported areas of “high need” for education research and/or technical 
assistance among state administrators in the Pacific region were English language learners 
(87%) and support for low-achieving schools (87%). 

Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the six largest 
percentages of state administrators in the Pacific region indicated “high need” were: 

■ English language learners (87%); 

■ Support for low- achieving schools (87%); 

■ Achievement gaps (83%); 

■ Teacher/ staff evaluation (77%); 


49 Results for the nation are presented in Chapter 3. 
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Details on the need for other areas of research and technical assistance are provided in 
Table 10-3. 


Sixty percent of state administrators in the Pacific region reported that their education 
research and technical assistance needs were met “very well” (as opposed to “moderately 
well” or “not well”), taking into account all sources of such research and technical 
assistance. 


Forty percent of state administrators in the Pacific region reported that their education 
research and technical assistance needs were met “moderately well” and 0 percent 
reported that their needs were “not well” met by their sources of assistance (Table 10- 

4). 


What sources of education research and technical assistance did state 
administrators use? 


The most reported source of education research and/or technical assistance for state 
administrators was professional associations (90%). Forty-four percent of state 
administrators in the Pacific region reported that they relied on the REL program “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance. 

■ State administrators in the Pacific region reported that they used a variety of sources for 
meeting their research and/ or technical assistance needs. They were most likely to rely 
“to a great extent” or “to a moderate extent” (as opposed to a “small extent” or not at 
all) on professional associations (90%) and colleges and universities (83%) (Figure 10-1). 

■ Forty-four percent of state administrators in the Pacific region reported that they relied 
on the REL program nationwide “to a great extent” or “to a moderate extent” for 
research and/or technical assistance (Figure 10-1). 
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Table 10-3. Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Pacific: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

Need for research and/or technical assistance 


High 

Moderate 

Low or no need 

n 

% 

% 

% 

Achievement gaps 

12 

83 

t 

t 

Assessment (formative or summative) 

12 

41 

t 

t 

Behavior, character education, or health 

12 

31 

t 

t 

College or career readiness 

12 

56 

44 

0 

Content standards, curriculum or instruction in STEM 

12 

60 

t 

t 

Content standards, curriculum or instruction in reading/writing 

12 

51 

t 

t 

Content standards, curriculum or instruction in other areas 

12 

37 

t 

t 

Dropout prevention 

12 

53 

47 

0 

Early childhood 

12 

37 

t 

t 

English language learners 

12 

87 

t 

t 

High school reform 

12 

37 

63 

0 

Leadership 

12 

54 

t 

t 

Longitudinal data systems 

12 

54 

t 

t 

Parental involvement 

12 

67 

33 

0 

Professional development 

12 

68 

t 

t 

Rural schools 

12 

58 

t 

t 

School accountability 

12 

51 

t 

t 

School choice 

12 

20 

27 

53 

School finance 

12 

41 

27 

32 

Students with disabilities 

12 

64 

t 

t 

Supplemental education services 

12 

51 

t 

t 

Support for low-achieving schools 

12 

87 

t 

t 

Teacher/staff evaluation 

12 

77 

t 

t 

Using data for decisions 

12 

68 

t 

t 

Other 

12 

0 

0 

0 


NOTE: Shaded cells are those that are mentioned in the text. % Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Table 10-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Pacific: School year 2011-12 


How well needs were met 

State administrators 

n 

% 

Very well 

7 

60 

Moderately well 

5 

40 

Not well 

0 

0 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Figure 10-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Pacific: School year 2011-2012 
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Comprehensive centers 
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NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on “other sources” of education research and/or technical 
assistance. The total N for state administrators was 12. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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■ Fifty percent of state administrators in the Pacific region reported that it was “very 

easy” or “moderately easy” to access education research and technical assistance across 
the available sources of information; and 50 percent reported it was “not at all easy” to 
access such assistance (Table 10-5). 

Table 10-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Pacific: School year 2011-12 


Ease of access 

State administrators 

n 

% 

Very easy or moderately easy to access 

6 

50 

Not at all easy to access 

6 

50 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state administrators with the REL program? 


Sixty three percent of state administrators in the Pacific region reported being “very 
familiar” or “somewhat familiar” with the REL program. 


■ Thirty-seven percent of state administrators in the Pacific region were either “a little 
familiar” or “not familiar at all” with the REL program (Table 10-6) 

Table 10-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all” with the REL program 
overall —Pacific: School year 2011-12 


Familiarity 

State administrators 

n 

% 

Very familiar 

3 

27 

Somewhat familiar 

5 

36 

A little familiar or not familiar at all 

4 

37 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How many state administrators used REL services? 


Seventy percent of state administrators in the Pacific region who were at least “a. little 
familiar” with the REL program reported that they used one or more REL services in the 
past 12 months. " 

Figure 10-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Pacific: School year 2011-12 
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NOTE: The total N for state administrators was 11. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Of state administrators in the Pacific region who were at least “a little familiar” with the 
REL program, 44 percent reported that they received technical assistance in the past 12 
months; 30 percent obtained information from the REL’s website; 26 percent attended 
a live or virtual event in the; and 23 percent received a response from a data or research 
request (Figure 10-2). 


50 Unless otherwise specified, the term ‘ at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” 
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In addition to asking respondents about the services they had used, state administrators in the 
Pacific region who had used REL services were also asked about the types of contact they had in the 
past 12 months with the REL. 

■ Among state administrators in the Pacific region who were at least “a little familiar” with 
the REL program and had used REL Pacific services in the past year, 53 percent said 
that they or their organization contacted the REL for research or other assistance; 43 
percent said they attended a REL-sponsored conference, training, or workshop; 43 
percent said they attended a meeting at which a REL representative was present; and 37 
percent said they forwarded someone else’s request to the REL (Table 10-7). 


Table 10-7. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Pacific: 
School year 2011-12 


Contact 

State administrators 

n 

% 

Contacted a reference desk or used the Ask a REL link on the website 

X 

X 

Attended a REL-sponsored conference, training, or workshop 

3 

43 

A REL representative was present at a meeting or workshop 

3 

43 

Contacted REL for research or other assistance 

4 

53 

Forwarded a request to the REL 

3 

37 

Other type of contact 

X 

X 


NOTE: The total N for state administrators was 8. $ Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state administrators? 


Eighty-four percent of state administrators in the Pacific region who were at least “ a little 
familiar” with the REL program overall were “somewhat satisfied” with it (Table 10-8). 
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Table 10-8. Percentage of the region’s administrators who were at least “a little familiar” with 
the REL program who were “very satisfied,” “somewhat satisfied," or “not at all 
satisfied” with it— Pacific: School year 2011-12 


Satisfaction 

State administrators 

n 

% 

Very satisfied 

X 

X 

Somewhat satisfied 

7 

84 

Not at all satisfied 

X 

X 


NOTE: t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Regional Educational Laboratory Southeast serves the following states: 

■ Alabama; 

■ Florida; 

■ Georgia; 

■ Mississippi; 

■ North Carolina; and 

■ South Carolina. 

For the 2006-1 1 contract period, REL Southeast was a university-based research center housed in 
the SERVE Center at the University of North Carolina — Greensboro. The SERVE Center had 
held the REL Southeast contract for the previous four cycles of funding. Florida State University 
was awarded the REL Southeast contract beginning in FY 2012. 


What were the technical quality and relevance of REL Southeast impact study 
reports published by IES and of the corresponding proposals ? 51 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 2011, IES published one impact study from REL 
Southeast: 


■ Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten. 


51 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 


Evaluation of the Regional Educational Laboratories: Final Report 

139 



REL: Southeast 


11 


The Kindergarten PAVEd for Success program is a 24- week in-class supplement to a school’s core 
language arts program. It is built around the three components of explicit vocabulary instruction, 
interactive book reading, and adult-child conversations. The study assessed the Kindergarten PAVEd 
for Success program’s effectiveness by comparing the expressive vocabulary and listening 
comprehension of students in the treatment and comparison groups at the end of the school year. 

Kindergarten students in schools using Kindergarten PAVEd for Success as a supplement to regular 
literacy instruction performed better than kindergarten students in comparison schools. The authors 
reported that students who received Kindergarten PAVEd for Success instruction were 1 month ahead 
in vocabulary development at the end of kindergarten, compared with students in the comparison 
group. 


The average quality ratings for the impact study report and proposal from REL Southeast 
that were reviewed by the expert panel were 4.51 and 3.96, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report quality rating fell 
between “strong” and “very strong,” and the average proposal quality rating fell between 
“adequate” and “strong” (Table 11-1). 


Table 11-1. Expert panel quality and relevance ratings for lES-published impact study reports 
and corresponding proposals from REL Southeast (on a 5-point scale with 5 being 
the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

3.96 

4.00 

lES-published impact study report 

4.51 

4.56 


Table Reads: For the proposal Effectiveness of a Program to Accelerate Vocabulary Development In Kindergarten, the mean quality 
dimension rating was 3.96. 


NOTE: The mean quality rating for proposals for REL Southeast was based on 27 indicator-specific ratings, and the mean relevance 
rating for proposals for REL Southeast was based on 15 indicator-specific ratings. The mean quality rating for reports for REL 
Southeast was based on 45 indicator-specific ratings, and the mean relevance rating for reports for REL Southeast was based on 18 
indicator-specific ratings. 

SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


The average relevance ratings for the impact study report and proposal from REL Southeast 
that were reviewed by the expert panel were 4.56 and 4.00, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report and proposal 
relevance ratings fell between “relevant” and “very relevant” (Table 11-1). 
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Table 11-2 displays, for each indicator of quality and relevance, the mean ratings from the expert 
panel review of the IES-published impact study and corresponding proposal from REL Southeast. 

Table 11-2. Mean ratings from expert panel review of impact studies for REL Southeast, by 
rating indicator 


Indicators for proposals 

Proposals 
(/V= 1) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.00 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

4.33 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

4.00 

ID. The design for random assignment is rigorous. 

3.67 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.33 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.67 

1G. The data collection plan is appropriate for the research questions. 

4.33 

1H. The data analyses will use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, multiple 
comparisons, and/or baseline non-equivalence between groups). 

4.33 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

5.00 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to meet 
the needs of the region served by the REL. 

4.33 

2B. The proposal provides a thorough summary of key literature and/or previous research in 
the topic area. 

3.67 

2C. The proposal provides a strong justification for selecting the particular intervention that is 
being studied. 

3.67 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.00 

2E. The proposal is clearly written and well presented. 

4.33 
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Table 11-2. Mean ratings from expert panel review of impact studies for REL Southeast, by 
rating indicator (continued) 


Indicators for reports 

lES-published 

reports 

(/V=l) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.67 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

4.67 

1C. There is minimal contamination in the form of crossover between subjects in 

treatment and control condition or spillover of the intervention from the treatment 
to the control group. 

5.00 

ID. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.67 

IE. The design and implementation of random assignment are rigorous. 

4.00 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

4.67 

1G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 

4.33 

1H. The data collection plan is appropriate for the research questions. 

4.33 

11. The data collection plan is well implemented. 

4.67 

U. The overall attrition rate and differential attrition rates are acceptable given the 
length of the intervention. 

4.67 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

5.00 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

5.00 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

4.67 

IN. All of the research questions are specifically addressed by the analyses. 

4.67 

10. The limitations of the study are clearly and comprehensively stated. 

3.67 

Relevance 

2A. The report provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

5.00 

2B. The report provides a thorough summary of key literature and/or previous research 
in the topic area. 

4.67 

2C. The report provides a strong justification for selecting the particular intervention that 
is being studied. 

4.00 

2D. The report contributes new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.33 

2E. The executive summary of the report is easy to read and understand for a lay 
audience. 

4.67 

2F. The report is clear and well written for the technical audience. 

4.67 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 
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How relevant and useful were the REL Southeast technical assistance products 
to the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Southeast technical assistance products were 
in meeting the needs of administrators in the Southeast region. State and district administrators were 
included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Southeast. 52 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Southeast region was support for low- 
achieving schools (57%). The most commonly reported area of “high need” for education 
research and/or technical assistance among district administrators in the Southeast region 
was content standards, curriculum, or instruction in areas other than reading/ writing or 
STEM (43%). 


52 Results for the nation are presented in Chapter 3. 
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Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the five largest 
percentages of state administrators in the Southeast region indicated “high need” were: 

■ Support for low-achieving schools (57%); 

■ Using data for decisions (47%); 

■ College or career readiness (47%); 

■ Achieving gaps (45%); and 

■ Teacher/ staff evaluation (45%). 

In the Southeast region, the five topic areas with the largest percentage of district administrators 
reporting “high need” were: 

■ Content standards, curriculum, or instmction in areas other than reading/ writing, or 
STEM (43%); 

■ Achievement gaps (40%); 

■ College or career readiness (35%); 

■ Dropout prevention (34%); and 

■ Content standards, curriculum or instmction in reading/writing (34%). 

Detail on the need for other areas of research and/ or technical assistance is provided in Table 11-3. 
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Table 11-3. Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Southeast: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

37 

45 

t 

r t 

454 

40 

45 

15 

Assessment (formative or summative) 

36 

41 

50 

9 

453 

27 

49 

23 

Behavior, character education, or health 

36 

11 

34 

54 

453 

16 

42 

42 

College or career readiness 

37 

47 

37 

16 

453 

35 

44 

21 

Content standards, curriculum or instruction in STEM 

36 

31 

53 

16 

453 

43 

42 

15 

Content standards, curriculum or instruction in reading/writing 

36 

28 

56 

17 

452 

34 

49 

17 

Content standards, curriculum or instruction in other areas 

36 

14 

56 

30 

450 

14 

52 

34 

Dropout prevention 

36 

44 

47 

9 

453 

34 

41 

25 

Early childhood 

36 

35 

30 

34 

453 

21 

38 

42 

English language learners 

36 

33 

56 

11 

450 

28 

38 

34 

High school reform 

36 

36 

36 

27 

453 

28 

41 

31 

Leadership 

35 

43 

38 

20 

454 

22 

46 

31 

Longitudinal data systems 

36 

20 

47 

33 

454 

25 

45 

30 

Parental involvement 

37 

25 

48 

27 

452 

30 

44 

25 

Professional development 

37 

27 

59 

14 

454 

26 

50 

25 

Rural schools 

37 

30 

49 

21 

453 

22 

32 

47 

School accountability 

36 

34 

38 

28 

453 

20 

49 

32 

School choice 

36 

8 

51 

42 

452 

5 

30 

64 

School finance 

36 

14 

31 

55 

452 

16 

39 

45 

Students with disabilities 

36 

34 

33 

33 

453 

27 

49 

24 

Supplemental education services 

36 

11 

45 

44 

454 

14 

37 

48 

Support for low-achieving schools 

37 

57 

30 

14 

454 

29 

38 

33 

Teacher/staff evaluation 

36 

45 

45 

11 

454 

21 

46 

33 

Using data for decisions 

36 

47 

37 

16 

452 

32 

46 

22 

Other 

4 

0 

0 

100 

44 

32 

20 

48 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. % Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Forty-eight percent of state administrators and 30 percent of district administrators reported 
that their education research and/ or technical assistance needs were met “very well” (as 
opposed to “moderately well” or “not well”), taking into account all sources of such 
research and technical assistance. 


■ Sixty-three percent of district administrators in the Southeast region reported that their 
education research and technical assistance needs were met “moderately well” and 8 
percent reported that their needs were “not well” met by their sources of assistance 
(Table 11-4). 

Table 11-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Southeast: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

17 

48 

132 

30 

Moderately well 

t 

t 

284 

63 

Not well 

* 

t 

35 

8 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance do state and 
district administrators use? 


The most reported sources of education research and/or technical assistance for state 
administrators was counterparts in other states and districts (91%) and professional 
associations (89%). For district administrators, the most reported sources of research 
and/ or technical assistance were counterparts in other states and districts (85%) and 
journals and publications (74%). Forty-nine percent of state administrators and 18 percent of 
district administrators in the Southeast region reported that they relied on the REL program 
“to a great extent” or “to a moderate extent” for research and/ or technical assistance. 

■ State and district administrators in the Southeast region reported that they used a variety 
of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on counterparts in other states and districts 
(91%) and professional associations (89%), while district administrators were most likely 
to rely “to a great extent” or “to a moderate extent” on counterparts in other states and 
districts (85%) and journals and publications (74%) (Figure 11-1). 
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Figure 11-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Southeast: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from ED, 
including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” The total Ns for state and 
district administrators on the item about reliance on the REL program were 37 and 451, respectively. The total N for state 
administrators on the items about other specified sources of research ranged from 36 to 37, depending on the number of 
respondents who chose not to respond to an individual item. The total N for state administrators for “other sources” was 5. The total N 
for district administrators on the items about other specified sources of research ranged from 453 to 454, depending on the number 
of respondents who chose not to respond to an individual item; and the total N for district administrators for “other sources” was 31. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Forty-nine percent of state administrators and 1 8 percent of district administrators in 
the Southeast region reported that they relied on the REL program nationwide “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance (Figure 

11 - 1 ). 


■ Forty-one percent of state administrators in the Southeast region reported that it was 
“very easy” (as opposed to “moderately easy” or “not at all easy”) to access education 
research and/or technical assistance across the available sources of information, and 39 
percent of district administrators in the Southeast region reported that it was “very 
easy” to access such assistance (Table 11-5). 
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Table 11-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research or technical assistance when 
needed— Southeast: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

15 

41 

179 

39 

Moderately easy to access 

16 

45 

252 

55 

Not at all easy to access 

5 

13 

23 

5 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Eighty-four percent of state administrators in the Southeast region reported being at least “ a 
little familiar” with the REL program, compared with 49 percent of district administrators. 


■ Sixteen percent of state administrators and 51 percent of district administrators in the 
Southeast region reported that they were “not familiar at all” with the REL program 
(Table 11-6). 

Table 11-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all” with the REL program 
overall— Southeast: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

10 

28 

15 

3 

Somewhat familiar 

12 

32 

79 

18 

A little familiar 

9 

24 

126 

28 

Not familiar at all 

6 

16 

235 

51 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey. 
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How many state and district administrators used REL services? 

Seventy-nine percent of state administrators and 45 percent of district administrators in the 
Southeast region who were at least u a little familiar” with the REL program reported that 
they used one or more REL services in the past 12 months. 53 

■ Majorities of state administrators in the Southeast region who were at least “a little 
familiar” with the REL program reported that they used each of two types of REL 
services in the past 12 months: responses to data or research requests (54%) and 
technical assistance (52%). Of district administrators in the Southeast region who were 
at least “a little familiar” with the REL program, 31 percent obtained information from 
the REL’s website, 1 7 percent attended a live or virtual event, 1 6 percent received a 
response from a data or research request, and 10 percent received technical assistance 
(Figure 11-2). 

Figure 11-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Southeast: School year 2011-12 


Used one or more services 


A live or virtual event 


Technical assistance 


Responses to data or research requests 


Information on the REL's website 


45 


49 

17 



79 

B State 
□ District 


0 20 40 60 80 100% 


NOTE: The total N for state administrators was 31; the total N for district administrators was 220. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey. 


53 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Southeast region. 
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■ Of administrators in the Southeast region who were at least “a little familiar” with the 
REL program 21 percent of state administrators and 55 percent of district 
administrators did not use any REL services in the past 12 months. 34 When asked why 
they had not used any REL services in the past year, the most common responses for 
state administrators (73 and 57 percent, respectively) were that they didn’t know what 
resources were available or they had no need for REL resources. When asked why they 
had not used any REL services in the past year, the most common responses for district 
administrators (63 and 55 percent, respectively) were that they didn’t know what 
services were available or their needs were met elsewhere (Table 11-7). 

Table 11-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Southeast: School year 2011-12 


Reason 

State administrators 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

X 

* 

37 

55 

Didn’t know what services were available 

5 

73 

43 

63 

Had no need for REL resources 

4 

57 

8 

12 

Not a good match between their current needs and the 
REL’s resources 

0 

0 

X 

* 

REL that served their state did not have a good 
reputation 

0 

0 

0 

0 


NOTE: The total N for state administrators was 7, and the total N for district administrators was 68. t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey. 


In addition to asking respondents about the services they had used, state and district administrators 
in the Southeast region who had used REL services were also asked about the types of contact they 
had in the past 12 months with the REL. 

■ Majorities of state administrators in the Southeast region who were at least “a little 

familiar” with the REL program and had used REL Southeast services in the past year 
indicated that they attended a meeting or workshop at which a REL representative was 
present (81%), or they or their organization contacted the REL for research or other 
assistance (67%). Among district administrators in the Southeast region who were at 
least “a little familiar” with the REL program and had used REL Southeast services in 
the past year, 32 percent said they attended a meeting at which a REL representative 
was present; 31 percent said that they or their organization contacted the REL for 
research or other assistance; 28 percent said they attended a REL-sponsored 
conference, training, or workshop; 28 percent contacted a reference desk for help or 
used the “Ask a REL” link on the REL’s website; and 15 percent had other types of 
contact with the REL (Table 11-8). 


54 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 11-8. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Southeast: 
School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

3 

12 

25 

28 

Attended a REL-sponsored conference, training, or 
workshop 

12 

49 

24 

28 

A REL representative was present at a meeting or 
workshop 

19 

81 

28 

32 

Contacted REL for research or other assistance 

16 

67 

27 

31 

Forwarded a request to the REL 

7 

30 

9 

10 

Other type of contact 

4 

16 

13 

15 


NOTE: The total N for state administrators was 24, and the total N for district administrators was 97. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey. 


How satisfied with the REL program were state and district administrators? 


Sixty-five percent of state administrators and 29 percent of district administrators in the 
Southeast region who were at least'''' a little familiar” with the REL program overall were 
“very satisfied” with it. 


■ Of the district administrators in the Southeast region who were at least “a little familiar” 
with the REL program nationwide, 58 percent reported being “somewhat satisfied” 
with it; and 12 percent reported being “not at all satisfied” with it (Table 11-9) 

Table 11-9. Percentage of the region’s administrators who were at least “a little familiar” with 
the REL program who were “very satisfied,” “somewhat satisfied,” or “not at all 
satisfied” with it— Southeast: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

17 

65 

50 

29 

Somewhat satisfied 

t 

t 

100 

58 

Not at all satisfied 

* 

t 

21 

12 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey. 
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Regional Educational Laboratory Southwest serves the following states: 

■ Arkansas; 

■ Louisiana; 

■ New Mexico; 

■ Oklahoma; and 

■ Texas. 

For the 2006-1 1 contract period, REL Southwest was headquartered at Edvance Research, Inc. in 
San Antonio, Texas. This was the first REL contract that Edvance Research had held. The REL 
Southwest contract beginning in FY 2012 was awarded to SEDL, which had held previous REL 
contracts. 


What were the technical quality and relevance of REL Southwest impact study 
reports published by IES and of the corresponding proposals ? 55 

As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 
quality and relevance of IES-published impact study reports and the corresponding proposals. 
Between March 1, 2006, and September 1, 2011, IES published one impact study from REL 
Southwest: 


■ The Impact of Collaborative Strategic Reading on the Reading Comprehension of Grade 5 Students in 
Tinguisticallj Diverse Schools. 


55 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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Collaborative Strategic Reading fCSRJ is a set of instructional strategies designed to improve the reading 
comprehension of students with diverse abilities. Teachers implement CSR at the classroom level 
using scaffolded instruction to guide students in the independent use of four comprehension 
strategies; students apply the strategies to informational text while working in small cooperative 
learning groups. The study did not find any impact from CSR on student reading comprehension. 


The average quality ratings for the impact study report and proposal from REL Southwest 
that were reviewed by the expert panel were 3.87 and 4.04, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report quality rating fell 
between “adequate” and “strong,” and the average proposal quality rating fell between 
“strong” and “very strong” (Table 12-1). 

Table 12-1. Expert panel quality and relevance ratings for lES-published impact study reports 
and corresponding proposals from REL Southwest (on a 5-point scale with 5 being 
the highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposal 

4.04 

4.07 

lES-published impact study report 

3.87 

4.00 


Table Reads: For the proposal The Impact of Collaborative Strategic Reading on the Reading Comprehension of Grade 5 Students in 
Linguistically Diverse Schools, the mean quality dimension rating was 4.04. 


NOTE: The mean quality rating for proposals for REL Southwest was based on 27 indicator-specific ratings, and the mean relevance 
rating for proposals for REL Southwest was based on 15 indicator-specific ratings. The mean quality rating for reports for REL 
Southwest was based on 45 indicator-specific ratings, and the mean relevance rating for reports for REL Southwest was based on 18 
indicator-specific ratings. 

SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


The average relevance ratings for the impact study report and proposal from REL 
Southwest that were reviewed by expert panels were 4.00 and 4.07, respectively. On a 5-point 
scale (1-5, with 1 being the lowest rating and 5 the highest), the average report relevance 
rating was “relevant,” and the average proposal relevance rating fell between “relevant” and 
“highly relevant” (Table 12-1). 

Table 12-2 displays, for each indicator of quality and relevance, the mean ratings from the expert 
panel review of the IES-published impact study and corresponding proposal from REL Southwest. 
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Table 12-2. Mean ratings from expert panel review of impact studies for REL Southwest, by 
rating indicator 


Indicators for proposals 

Proposals 

(/v=D 


Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 


IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

3.67 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

4.33 

ID. The design for random assignment is rigorous. 

4.33 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.67 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.00 

1G. The data collection plan is appropriate for the research questions. 

4.33 

1H. The data analyses will use appropriate methods to address the research questions 
(e.g., adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

5.00 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

4.67 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.00 


2B. The proposal provides a thorough summary of key literature and/or previous research 
in the topic area. 

3.67 

2C. The proposal provides a strong justification for selecting the particular intervention 
that is being studied. 

4.33 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.00 

2E. The proposal is clearly written and well presented. 

4.33 
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Table 12-2. Mean ratings from expert panel review of impact studies for REL Southwest, by 
rating indicator (continued) 


Indicators for reports 
Quality 

lES-published 

reports 

(/V=l) 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.33 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

3.00 

1C. There is minimal contamination in the form of crossover between subjects in 

treatment and control condition or spillover of the intervention from the treatment to 
the control group. 

3.67 

ID. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.67 

IE. The design and implementation of random assignment are rigorous. 

3.67 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

3.67 

1G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 

3.33 

1H. The data collection plan is appropriate for the research questions. 

3.67 

11. The data collection plan is well implemented. 

3.67 

U. The overall attrition rate and differential attrition rates are acceptable given the 
length of the intervention. 

3.67 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

4.33 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.33 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

4.33 

IN. All of the research questions are specifically addressed by the analyses. 

4.67 

10. The limitations of the study are clearly and comprehensively stated. 

Relevance 

4.00 

2A. The report provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.67 

2B. The report provides a thorough summary of key literature and/or previous research 
in the topic area. 

4.00 

2C. The report provides a strong justification for selecting the particular intervention that 
is being studied. 

3.67 

2D. The report contributes new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

3.33 

2E. The executive summary of the report is easy to read and understand for a lay 
audience. 

4.00 

2F. The report is clear and well written for the technical audience. 

4.33 


NOTE: The mean for each quality and relevance indicator was based on three ratings. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 


Evaluation of the Regional Educational Laboratories: Final Report 

156 



REL: Southwest 


12 


How relevant and useful were the REL Southwest technical assistance products 
to the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL Southwest technical assistance products were 
in meeting the needs of administrators in the Southwest region. State and district administrators 
were included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL Southwest. 56 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the Southwest region was college or career 
readiness (60%). The most commonly reported area of “high need” for education research 
and/ or technical assistance among district administrators in the Southwest region was 
achievement gaps (38%). 


56 Results for the nation are presented in Chapter 3. 
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Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the eight largest 
percentages of state administrators in the Southwest region indicated “high need” were: 

■ College or career readiness (60%); 

■ Teacher/ staff evaluation (57%); 

■ Content standards, curriculum, or instmction in areas other than reading/ writing or 
STEM (53%); 

■ Dropout prevention (48%); 

■ Assessment (43%) 

■ Content standards, curriculum or instmction in reading/ writing (43%); 

■ Professional development (43%); and 

■ Rural schools (43%). 

In the Southwest region, the five topic areas with the largest percentage of district administrators 
reporting “high need” were: 

■ Achievement gaps (38%); 

■ Content standards, curriculum, or instmction in areas other than reading/ writing or 
STEM (37%); 

■ College or career readiness (33%); 

■ Content standards, curriculum, or instmction in reading/writing (31%); and 

■ Using data for decisions (29%). 

Detail on the need for other areas of research and technical assistance is provided in Table 12-3. 
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Table 12-3. Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— Southwest: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

21 

38 

48 

15 

438 

38 

44 

18 

Assessment (formative or summative) 

21 

43 

20 

37 

438 

26 

45 

29 

Behavior, character education, or health 

20 

15 

30 

54 

437 

18 

48 

34 

College or career readiness 

21 

60 

19 

20 

435 

33 

45 

22 

Content standards, curriculum or instruction in STEM 

21 

53 

24 

23 

437 

37 

45 

18 

Content standards, curriculum or instruction in reading/writing 

21 

43 

34 

23 

437 

31 

45 

24 

Content standards, curriculum or instruction in other areas 

21 

t 

t 

33 

435 

12 

50 

38 

Dropout prevention 

21 

48 

24 

28 

435 

25 

42 

32 

Early childhood 

21 

33 

20 

47 

438 

18 

42 

40 

English language learners 

21 

38 

48 

14 

437 

28 

43 

30 

High school reform 

21 

39 

38 

23 

438 

25 

42 

33 

Leadership 

21 

38 

t 

t 

438 

22 

53 

25 

Longitudinal data systems 

21 

29 

52 

19 

438 

20 

50 

30 

Parental involvement 

21 

38 

34 

28 

438 

28 

48 

25 

Professional development 

21 

43 

t 

t 

438 

24 

55 

21 

Rural schools 

21 

43 

29 

28 

437 

24 

30 

46 

School accountability 

21 

29 

48 

24 

436 

20 

48 

32 

School choice 

21 

19 

34 

47 

436 

8 

28 

64 

School finance 

20 

30 

44 

25 

437 

22 

39 

39 

Students with disabilities 

20 

30 

40 

30 

438 

22 

50 

28 

Supplemental education services 

21 

25 

61 

14 

437 

13 

49 

38 

Support for low-achieving schools 

21 

42 

44 

14 

438 

28 

40 

33 

Teacher/staff evaluation 

21 

57 

19 

24 

439 

22 

43 

34 

Using data for decisions 

21 

42 

t 

t 

438 

29 

47 

24 

Other 

t 

t 

t 

t 

48 

23 

19 

58 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. X Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Twenty-four percent of state administrators and 30 percent of district administrators in the 
Southwest region reported that their education research and / or technical assistance needs 
were met “very well” (as opposed to “moderately well” or “not well”), taking into account 
all sources of such research and technical assistance. 


■ Sixty percent of district administrators in the Southwest region reported that their 

education research and technical assistance needs were met “moderately well” and 10 
percent reported that their needs were “not well” met by their sources of assistance,” 
(Table 12-4). 

Table 12-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well,” or “not well," taking into 
account all sources of assistance— Southwest: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

5 

24 

130 

30 

Moderately well 

t 

t 

265 

60 

Not well 

t 

* 

43 

10 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported source of education research and/ or technical assistance for state and 
district administrators was counterparts in other SEAs and LEAs (82 and 83 percent, 
respectively). Twenty-seven percent of state administrators and 18 percent of district 
administrators in the Southwest region reported that they relied on the REL program “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance. 

■ State and district administrators in the Southwest region reported that they used a 
variety of sources for meeting their research and/ or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on counterparts in other states and districts 
(82%) and consulting firms or contractors (70%), while district administrators were 
most likely to rely “to a great extent” or “to a moderate extent” on counterparts in other 
states and districts (83%) and journals and publications (68%) (Figure 12-1). 
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Figure 12-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— Southwest: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from ED, 
including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other Sources 



27 


28 


35 


37 


66 


57 


67 


□ State 

□ District 



20 


40 


60 


80 


100% 


NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied to a great or moderate extent on “other sources" of education research and/or technical assistance. The total 
Ns for state and district administrators on the item about reliance on the REL program were 16 and 188, respectively. The total N for 
state administrators on the items about other specified sources of research ranged from 20 to 21. The total N for district 
administrators on the items about other specified sources of research ranged from 435 to 437, depending on the number of state 
and district respondents who chose not to respond to an individual item; and the total N for district administrators for “other sources” 
was 32. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Twenty-seven percent of state administrators and 1 8 percent of district administrators 
in the Southwest region reported that they relied on the REL program nationwide “to a 
great extent” or “to a moderate extent” for research and/ or technical assistance (Figure 
12 - 1 ). 

■ Thirty-two percent of state administrators and 37 percent of district administrators in 
the Southwest region reported that it was “very easy” (as opposed to “moderately easy” 
or “not at all easy”) to access education research and/ or technical assistance across the 
available sources of information. (Table 12-5). 
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Table 12-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— Southwest: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

7 

32 

163 

37 

Moderately easy to access 

10 

48 

244 

56 

Not at all easy to access 

4 

19 

31 

7 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Seventy-one percent of state administrators in the Southwest region reported being at least 
“a little familiar” with the REL program, compared with 44 percent of district 
administrators. 


■ Twenty-nine percent of state administrators and 56 percent of district administrators in 
the Southwest region report that they were “not familiar at all” with the REL program 
(Table 12-6). 

Table 12-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all” with the REL program 
overall —Southwest: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

X 

X 

12 

3 

Somewhat familiar 

X 

X 

71 

16 

A little familiar 

8 

39 

112 

25 

Not familiar at all 

6 

29 

245 

56 


NOTE: Percentages may not sum to 100 due to rounding, t Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Evaluation of the Regional Educational Laboratories: Final Report 

162 



REL: Southwest 


12 


How many state and district administrators used REL services? 

Fifty-eight percent of state administrators and 38 percent of district administrators in the 
Southwest region who were at least 11 a little familiar” with the REL program reported that 
they used one or more REL services in the past 12 months. 57 

■ State administrators in the Southwest region who were at least “a little familiar” with the 
REL program reported that in the past 12 months they were most likely to have used 
responses from the REL to data or research requests (34%). Of district administrators 
in the Southwest region who were at least “a little familiar” with the REL program, 30 
percent obtained information from the REL’s website; 18 percent attended a live or 
virtual event; 9 percent received a response from a data or research request; and 9 
percent received technical assistance (Figure 12-2). 

■ Of administrators in the Southwest region who were at least “a little familiar” with the 
REL program 37 percent of state administrators and 35 percent of district 
administrators did not use any REL services in the past 12 months. 35 When asked why 
they had not used any REL services in the past year, the most common response for 
state administrators was that their needs were met elsewhere (79 percent). The most 
common responses for district administrators (56 and 54 percent, respectively) were 
that their needs were met elsewhere or they didn’t know what services were available 
(Table 12-7). 


57 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the Southwest region. 

58 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Figure 12-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
Southwest: School year 2011-12 


Used one or more services 


A live or virtual event 


Technical assistance 


Responses to data or research requests 
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NOTE: The total N for state administrators was 15, and the total N for district administrators was 195. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Table 12-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— Southwest: School year 2011-12 


Reason 

State administrators 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

4 

79 

38 

56 

Didn’t know what services were available 

0 

0 

37 

54 

Had no need for REL resources 

0 

0 

12 

18 

Not a good match between their current needs and the 
REL’s resources 

0 

0 

7 

10 

REL that served their state did not have a good 
reputation 

0 

0 

3 

4 


NOTE: The total N state administrators was 5, and the total N for district administrators was 68. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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In addition to asking respondents about the services they had used, state and district administrators 
in the Southwest region who had used REL services were also asked about the types of contact they 
had in the past 1 2 months with the REL. 


■ Majorities of state administrators in the Southwest region who were at least “a litde 

familiar” with the REL program and had used REL Southwest services in the past year 
indicated that they or their organization contacted the REL for research or other 
assistance (77%), they attended a REL-sponsored conference, training or workshop 
(54%), or they attended a meeting or workshop at which a REL representative was 
present (33%). Among district administrators in the Southwest region who were at least 
“a little familiar” with the REL program and had used REL Southwest services in the 
past year, 42 percent said that they or their organization contacted the REL for research 
or other assistance; 35 percent contacted a reference desk for help or used the Ask a 
REL link on the REL’s website; 31 percent said they attended a REL-sponsored 
conference, training, or workshop; 28 percent said they attended a meeting at which a 
REL representative was present; and 18 percent had other types of contact (Table 12-8). 

Table 12-8. Percentage of administrators who had used REL services and reported having 

various types of contact with the REL serving their state in the past year— Southwest: 
School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

t 

t 

22 

35 

Attended a REL-sponsored conference, training, or 
workshop 

5 

54 

19 

31 

A REL representative was present at a meeting or 
workshop 

3 

33 

17 

28 

Contacted REL for research or other assistance 

7 

77 

26 

42 

Forwarded a request to the REL 

t 

t 

t 

t 

Other type of contact 

* 

$ 

11 

18 


NOTE: The total N state administrators was 9, and the total N for district administrators was 66. ^ Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 


Forty-nine percent of state administrators and 26 percent of district administrators in the 
Southwest region who were familiar with the REL program nationwide were “very satisfied” 
with it. 


■ Of the state administrators in the Southwest region who were at least “a little familiar” 
with the REL program overall, 52 percent reported being “somewhat satisfied” with it 
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and 0 percent of state administrators reported being “not at all satisfied” with it (Table 
12-9). 


Table 12-9. Percentage of the region’s administrators who were at least “a little familiar” with 
the REL program who were “very satisfied,” “somewhat satisfied,” or “not at all 
satisfied” with it— Southwest: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

5 

48 

39 

26 

Somewhat satisfied 

5 

52 

88 

58 

Not at all satisfied 

0 

0 

24 

16 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Of the district administrators in the Southwest region who were at least “a little familiar” 
with the REL program overall, 58 percent reported being “somewhat satisfied” with it 
and 16 percent reported being “not at all satisfied” with it (Table 12-9). 
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Regional Educational Laboratory West serves the following states: 

■ Arizona, 

■ California, 

■ Nevada, and 

■ Utah. 

For the 2006-1 1 contract period, REL West was headquartered at WestEd in San Francisco, 
California. WestEd had held previous REL contracts and was also awarded the REL West contract 
beginning in FY 2012. 


As part of the evaluation of the RELs, Westat conducted an expert panel review to examine the 


Between March 1, 2006, and September 1, 2011, IES published two impact studies from REL West: 

■ Effects of Problem-Based Economics on High School Economics Instruction, and 

■ Accommodations for English Language Eearner Students: The Effect of Linguistic Modification of 


What were the technical quality and relevance of REL West impact study 
reports published by IES and of the corresponding proposals ? 59 


quality and relevance of IES-published impact study reports and the corresponding proposals. 


Math Test Item Sets. 


59 Impact studies are designed to make causal inferences about an intervention, policy, or practices, typically using RCTs 
or regression discontinuity designs. 
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Building on the more general techniques of problem-based learning, the Problem Based Economics 
Instruction curriculum was designed to help students actively learn critical thinking and problem- 
solving skills using real-world examples. The study assessed the effectiveness of Problem Based 
Economics Instruction by comparing the economics knowledge of students in the treatment and 
comparison groups at the end of their implementation semester. The study found no statistically 
significant difference on the economics knowledge of students in grades 1 1 and 12 in the classes that 
used Problem Based Economics Instruction , relative to students in the comparison classes. However, the 
effect size of 0.29 for the economics knowledge domain was positive. 

Einguistic modification is a test accommodation strategy aimed at removing language barriers. This 
strategy requires the creation of carefully constructed test items that are accessible to all students, 
regardless of language background, while still maintaining the integrity of the content being tested. 
The purpose of the study was to assess the effects of using linguistic modification as a way of removing 
language barriers for English language learners and non-English language learners (EL) struggling 
with reading. This study focuses on the linguistic modification of math content that is typically 
presented on standardized math achievement tests. The study found a positive effect on math scores 
for students struggling with English who completed the linguistic modification item set relative to 
similar students who did not. The estimated six percentage-point gain on math achievement is 
statistically significant. The study found neither statistically significant nor substantively important 
differences for EP students who took the modified test, relative to those who did not. 

The average quality ratings for the two impact study reports and two proposals from REL 
West that were reviewed by expert panels were 4.04 and 3.69, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report quality rating fell 
between “strong” and “very strong,” and the average proposal quality rating fell between 
“adequate” and “strong” 

The average relevance ratings for the two impact study reports and two proposals from REL 
West that were reviewed by expert panels were 4.25 and 3.87, respectively. On a 5-point scale 
(1-5, with 1 being the lowest rating and 5 the highest), the average report relevance rating 
fell between “relevant” and “very relevant,” and the average proposal relevance rating fell 
between “adequate” and “relevant” (Table 13-1). 

Table 13-2 displays, for each indicator of quality and relevance, the mean ratings from expert panel 
reviews of IES-published impact studies and corresponding proposals from REL West. 
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Table 13-1. Expert panel quality and relevance ratings for lES-published impact study reports 
and corresponding proposals from REL West (on a 5-point scale with 5 being the 
highest) 


Product 

Mean ratings 

Quality 

Relevance 

Impact study proposals 



Proposals from REL West ( N = 2) 

3.69 

3.87 

Effects of Problem Based Economics on High School Economics 
Instruction 

3.96 

3.73 

Accommodations for English Language Learner Students: The Effect of 
Linguistic Modification of Math Test Item Sets 

3.41 

4.00 

lES-published impact study reports 



Impact study reports from REL West ( N = 2) 

4.04 

4.25 

Effects of Problem-Based Economics on High School Economics 
Instruction 

3.98 

4.39 

Accommodations for English Language Learner Students: The Effect of 
Linguistic Modification of Math Test Item Sets 

4.11 

4.11 


Table Reads: For the 2 proposals for impact studies from REL West, the mean quality dimension rating was 3.69. 


NOTE: N = Number of lES-published reports prepared under the 2006-11 REL contracts and released by September 1, 2011, or 

corresponding proposals submitted by the RELs between March 1, 2006, and September 1, 2011, and reviewed by expert panels. The 
mean quality rating for proposals for REL West was based on 54 indicator-specific ratings, and the mean relevance rating for 
proposals for REL West was based on 30 indicator-specific ratings. The mean quality rating for reports for REL West was based on 90 
indicator-specific ratings, and the mean relevance rating for reports for REL West was based on 36 indicator-specific ratings. 
SOURCES: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 
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Table 13-2. Mean ratings from expert panel review of impact studies for REL West, by rating 
indicator 


Indicators for proposals 

Proposals 
(N= 2) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

3.00 

IB. The plan to document implementation of the intervention (e.g., exposure, quality of 
delivery, adherence) is appropriate. 

3.50 

1C. The research questions are explicitly stated, aligned with the study, and are empirically 
testable. 

4.50 

ID. The design for random assignment is rigorous. 

3.67 

IE. The sampling strategies are appropriate for targeted populations, and the resulting 
sample size(s) for the impact questions will have adequate statistical power. 

3.83 

IF. Outcome measures are valid, reliable, and not overly aligned with the intervention. 

3.50 

1G. The data collection plan is appropriate for the research questions. 

4.00 

1H. The data analyses will use appropriate methods to address the research questions 
(e.g., adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

3.33 

11. The proposed research is feasible to be carried out with reasonable timeline and 
sequencing of activities. 

3.83 

Relevance 

2A. The proposal provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

3.83 

2B. The proposal provides a thorough summary of key literature and/or previous research 
in the topic area. 

3.50 

2C. The proposal provides a strong justification for selecting the particular intervention that 
is being studied. 

3.33 

2D. The report will contribute new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.50 

2E. The proposal is clearly written and well presented. 

4.17 
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Table 13-2. Mean ratings from expert panel review of impact studies for REL West, by rating 
indicator (continued) 


Indicators for reports 

lES-published 
reports 
{N= 2) 

Quality 

1A. The intervention for the treatment group and the condition for the control group are 
clearly described. 

4.50 

IB. Implementation of the intervention is well documented (e.g., exposure, quality of 
delivery, adherence). 

3.83 

1C. There is minimal contamination in the form of crossover between subjects in 

treatment and control condition or spillover of the intervention from the treatment 
to the control group. 

4.50 

ID. The research questions are explicitly stated, aligned with the study, and are 
empirically testable. 

4.67 

IE. The design and implementation of random assignment are rigorous. 

3.83 

IF. The sampling strategies are appropriate for targeted populations and the resulting 
sample size(s) for the impact questions have adequate statistical power. 

3.50 

1G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 

3.83 

1H. The data collection plan is appropriate for the research questions. 

4.33 

11. The data collection plan is well implemented. 

4.33 

U. The overall attrition rate and differential attrition rates are acceptable given the 
length of the intervention. 

3.67 

IK. The data analyses use appropriate methods to address the research questions (e.g., 
adjusting where necessary for any clustering, missing data from nonresponse, 
multiple comparisons, and/or baseline non-equivalence between groups). 

3.67 

1L. Appropriate statistics are provided to describe the sample and support the findings. 

4.17 

1M. The conclusions about the intervention are drawn appropriately and consistently. 

3.67 

IN. All of the research questions are specifically addressed by the analyses. 

4.00 

10. The limitations of the study are clearly and comprehensively stated. 

4.17 

Relevance 

2A. The report provides a strong justification why the topic was selected as salient to 
meet the needs of the region served by the REL. 

4.50 

2B. The report provides a thorough summary of key literature and/or previous research 
in the topic area. 

4.33 

2C. The report provides a strong justification for selecting the particular intervention that 
is being studied. 

4.17 

2D. The report contributes new information about the effectiveness of the intervention 
being studied and the more general topic being addressed. 

4.17 

2E. The executive summary of the report is easy to read and understand for a lay 
audience. 

3.83 

2F. The report is clear and well written for the technical audience. 

4.50 


NOTE: The mean for each quality and relevance indicator was based on six ratings. 


SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
reports); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for 
proposals) (Appendix A). 
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How relevant and useful were the REL West technical assistance products to 
the needs of the states and districts in the region? 

Between the fall of 201 1 and spring of 2012, a survey of state and district administrators was 
conducted to determine how relevant and useful REL West technical assistance products were in 
meeting the needs of administrators in the West region. State and district administrators were 
included in the sample regardless of previous use of REL services or familiarity with the REL 
program. Specific research questions included: 

■ What needs did state and district administrators have for education research and 
technical assistance, and were those needs met? 

■ What sources of education research and technical assistance did state and district 
administrators use? 

■ How familiar were state and district administrators with the REL program? 

■ How many state and district administrators used REL services? 

■ How satisfied with the REL program were state and district administrators? 

This section presents the responses to those questions based on the results from the REL survey of 
state and district administrators in REL West. 6 " 


What needs did state and district administrators have for education research 
and technical assistance, and were those needs met? 

The most commonly reported area of “high need” for education research and/or technical 
assistance among state administrators in the West region was teacher/ staff evaluation 
(66%). The most commonly reported area of “high need” for education research and/or 
technical assistance among district administrators in the West region was using data for 
decisions (39%). 


60 Results for the nation are presented in Chapter 3. 
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Administrators were asked to indicate whether they had a “high need,” “moderate need,” or “low or 
no need” for research/ assistance in specific topic areas. The areas in which the five largest 
percentages of state administrators in the West region indicated “high need” were: 

■ Teacher/ staff evaluation (65%); 

■ English language learners (58%); 

■ Using data for decisions (56%); 

■ Assessment (52%), and 

■ College or career readiness (51%). 

In the West region, the eight topic areas with the largest percentage of district administrators 
reporting “high need” were: 

Using data for decisions (39%); 

Content standards, curriculum, or instmction in STEM (38%); 

English language learners (38%); 

Achievement gaps (35%); 

Assessment (34%); 

College or career readiness (34%); 

Longitudinal data systems (34%); and 
Support for low- achieving schools (34%). 

on the need for other areas of research and/ and/ or technical assistance is provided in Table 


Detail 

13-3. 
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Table 13-3. Percentage of all administrators who reported various levels of need for different types of research and technical 
assistance— West: School year 2011-12 


Type of research and/or technical assistance 

State administrators 

District administrators 

Need for research and/or technical assistance 

Need for research and/or technical assistance 


High 

Moderate 

Low or no 
need 


High 

Moderate 

Low or no 
need 

n 

% 

% 

% 

n 

% 

% 

% 

Achievement gaps 

20 

47 

t 

t 

393 

35 

47 

17 

Assessment (formative or summative) 

20 

52 

32 

16 

393 

34 

46 

21 

Behavior, character education, or health 

20 

15 

16 

70 

391 

21 

40 

39 

College or career readiness 

20 

51 

t 

t 

390 

34 

38 

29 

Content standards, curriculum or instruction in STEM 

20 

30 

56 

14 

392 

38 

42 

20 

Content standards, curriculum or instruction in reading/writing 

19 

16 

55 

29 

392 

28 

48 

24 

Content standards, curriculum or instruction in other areas 

19 

0 

56 

45 

391 

11 

47 

41 

Dropout prevention 

19 

33 

46 

20 

393 

26 

34 

40 

Early childhood 

19 

27 

t 

t 

390 

17 

36 

47 

English language learners 

19 

58 

t 

t 

393 

38 

36 

26 

High school reform 

19 

49 

31 

21 

391 

28 

33 

39 

Leadership 

19 

27 

54 

19 

392 

29 

46 

25 

Longitudinal data systems 

19 

49 

35 

15 

392 

34 

39 

26 

Parental involvement 

19 

25 

33 

42 

394 

23 

48 

29 

Professional development 

20 

41 

35 

24 

392 

31 

49 

20 

Rural schools 

19 

23 

53 

25 

392 

25 

25 

50 

School accountability 

19 

32 

53 

16 

392 

19 

45 

36 

School choice 

19 

* 

t 

68 

392 

8 

28 

64 

School finance 

19 

15 

55 

31 

389 

18 

36 

46 

Students with disabilities 

19 

t 

t 

37 

391 

25 

47 

28 

Supplemental education services 

19 

t 

t 

50 

392 

13 

44 

43 

Support for low-achieving schools 

20 

35 

51 

14 

393 

34 

36 

30 

Teacher/staff evaluation 

20 

65 

t 

t 

392 

30 

46 

24 

Using data for decisions 

20 

56 

t 

t 

393 

39 

41 

21 

Other 

t 

t 

t 

t 

48 

32 

18 

50 


NOTE: Percentages may not sum to 100 due to rounding. Shaded cells are those that are mentioned in the text. X Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratories (REL) Customer Survey, fall 2011 to spring 
2012 . 
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Thirty-five percent of state administrators and 27 percent of district administrators in the 
West region reported that their education research and/or technical assistance needs were 
met “very well” (as opposed to “moderately well” or “not well”), taking into account all 
sources of such research and technical assistance. 


■ Sixty-five percent of state administrators in the West region reported that their 

education research and/ or technical assistance needs were met “moderately well” and 0 
percent reported that their needs were “not well” met by their sources of assistance 
(Table 13-4). 

Table 13-4. Percentage of all administrators who reported that their research and technical 

assistance needs were met “very well," “moderately well," or “not well," taking into 
account all sources of assistance— West: School year 2011-12 


How well needs were met 

State administrators 

District administrators 

n 

% 

n 

% 

Very well 

7 

35 

106 

27 

Moderately well 

13 

65 

239 

62 

Not well 

0 

0 

43 

11 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Sixty-two percent of district administrators in the West region reported that their 

education research and technical assistance needs were met “moderately well” and 1 1 
percent reported that their needs were “not well” met by their sources of assistance 
(Table 13-4). 


What sources of education research and technical assistance did state and 
district administrators use? 


The most reported source of education research and/ or technical assistance for state 
administrators was professional associations (91%), and for district administrators it was 
counterparts in other states and districts (84%). Seventy-one percent of state administrators 
and 19 percent of district administrators in the West region reported that they relied on the 
REL program “to a great extent” or “to a moderate extent” for research and/or technical 
assistance. 


■ State and district administrators in the West region reported that they used a variety of 
sources for meeting their research and/or technical assistance needs. State 
administrators were most likely to rely “to a great extent” or “to a moderate extent” (as 
opposed to a “small extent” or not at all) on professional associations (91%) and 
counterparts in other states and districts (81%), while district administrators were most 
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likely to rely “to a great extent” or “to a moderate extent” on counterparts in other 
states and districts (84%) and journals and publications (74%) (Figure 13-1). 

■ Seventy-one percent of state administrators and 19 percent of district administrators in 
the West region reported that they relied on the REL program nationwide “to a great 
extent” or “to a moderate extent” for research and/or technical assistance (Figure 13-1). 

Figure 13-1. Percentage of all administrators who reported that they relied on different sources 
of education research and/or technical assistance “to a great extent” or “to a 
moderate extent”— West: School year 2011-12 


REL program nationwide 
Comprehensive centers 

Other federally funded providers 

Other products or resources from 
ED, including websites 

Professional associations 
Colleges and universities 
Consulting firms/contractors 
Counterparts at other LEAs or SEAs 
TA center supported by your state 
Journals and publications 
Other sources 



0 20 40 60 80 100 % 


NOTE: “Other products or resources from ED” was specified as “including websites such as Doing What Works.” No state administrators 
reported that they relied “to a great extent” or “to a moderate extent” on “other sources” of education research and/or technical 
assistance. The total Ns for state and district administrators on the item about reliance on the REL program were 18 and 195, 
respectively. The total N for state administrators on the items about other specified sources of research ranged from 1-20. The total N 
for district administrators on the items about other specified sources of research ranged from 389 to 392, depending on the number 
of respondents who chose not to respond to an individual item; and the total N for district administrators for “other sources” was 18. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


Evaluation of the Regional Educational Laboratories: Final Report 

176 



REL: West 


13 


■ Twenty-six percent of state administrators in the West region reported that it was “very 
easy” (as opposed to “moderately easy” or “not at all easy”) to access education research 
and/or technical assistance across the available sources of information, and 38 percent 
of district administrators in the West region reported that it was “very easy” to access 
such assistance (Table 13-5). 

Table 13-5. Percentage of all administrators who reported that it was “very easy,” “moderately 
easy,” or “not at all easy” to access education research and/or technical assistance 
when needed— West: School year 2011-12 


Ease of access 

State administrators 

District administrators 

n 

% 

n 

% 

Very easy to access 

5 

26 

151 

38 

Moderately easy to access 

15 

74 

206 

53 

Not at all easy to access 

0 

0 

35 

9 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How familiar were state and district administrators with the REL program? 


Seventy percent of state administrators in the West region reported being “very familiar” or 
“somewhat familiar” with the REL program, compared with 31 percent of district 
administrators. 


■ Twenty-four percent of district administrators in the West region were “a little familiar” 
with the REL program, and 45 percent of district administrators in the West region 
were “not familiar at all” with the REL program (Table 13-6). 

Table 13-6. Percentage of all administrators who reported that they were “very familiar,” 

“somewhat familiar,” “a little familiar,” or “not familiar at all” with the REL program 
overall —West: School year 2011-12 


Familiarity 

State administrators 

District administrators 

n 

% 

n 

% 

Very familiar 

6 

28 

33 

8 

Somewhat familiar 

8 

42 

90 

23 

A little familiar 

t 

t 

93 

24 

Not familiar at all 

t 

t 

178 

45 


NOTE: Percentages may not sum to 100 due to rounding. $ Reporting standards were not met. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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How many state and district administrators use REL services? 

Ninety-four percent of state administrators and 63 percent of district administrators in the 
West region who were at least “ a little familiar” with the REL program reported that they 
used one or more REL services in the past 12 months. 61 


■ Majorities of state administrators in the West region who were at least “a little familiar” 
with the REL program reported that they used each of three types of REL services in 
the past 12 months: technical assistance (82%), a live or virtual event (61%), and 
responses to data or research requests (59%). Of district administrators in the West 
region who were at least “ a little familiar” with the REL program, 38 percent attended a 
live or virtual event; 38 percent obtained information from the REL’s website; 21 
percent received technical assistance; and 18 percent received a response from a data or 
research request (Figure 13-2). 


61 Unless otherwise specified, the term ‘at least “a little familiar” with the REL program’ includes “very familiar,” 
“somewhat familiar,” or “a little familiar.” Note that administrators’ use of services was contingent on familiarity, 
which differed for states and districts in the West region. 
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Figure 13-2. Percentage of administrators who were at least “a little familiar” with the REL 

program who reported that they used various REL services in the past 12 months— 
West: School year 2011-12 


Used one or more services 


A live or virtual event 


38 


63 


61 


94 


OSState 
□ District 


Technical assistance 


21 


82 


Responses to data or research requests 


18 


59 


Information on the REL's website 


22 


38 


20 


40 


60 


80 


100 % 


NOTE: The total N for state administrators was 18, and the total N for district administrators was 216. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


■ Of administrators in the West region who were at least “a little familiar” with the REL 
program, 6 percent of state administrators and 25 percent of district administrators did 
not use any REL services in the past 12 months/’ 2 When asked why they had not used 
any REL services in the past year, the most common responses for district 
administrators (57 and 51 percent, respectively) were that their needs were met 
elsewhere or they didn’t know what services were available (Table 13-7). 


62 Percentages may not sum to 100 because some administrators did not know if they had used REL services. 
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Table 13-7. Reasons administrators who were at least “a little familiar” with the REL program 
did not use REL services in the past 12 months— West: School year 2011-12 


Reason 

State administrators 

District administrators 

n 

% 

n 

% 

Needs were met elsewhere 

t 

t 

30 

57 

Didn’t know what services were available 

t 

t 

27 

51 

Had no need for REL resources 

0 

0 

17 

32 

Not a good match between their current needs and the 
REL’s resources 

0 

0 

5 

9 

REL that served their state did not have a good 
reputation 

0 

0 

t 

t 


NOTE: The total N for district administrators was 53. t Reporting standards were not met. 

SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


In addition to asking respondents about the services they had used, state and district administrators 
in the West region who had used REL services were also asked about the types of contact they had 
in the past 1 2 months with the REL. 

■ Majorities of state administrators in the West region who were at least “a little familiar” 
with the REL program and had used REL West services in the past year indicated that: 
they or their organization contacted the REL for research or other assistance (93%); 
they attended a meeting or workshop at which a REL representative was present (82%); 
they attended a REL-sponsored conference, training, or workshop (52%); or they 
forwarded a request they had received to the REL (17 percent). Among district 
administrators in the West region who were at least “a little familiar” with the REL 
program and had used REL West services in the past year, 48 percent said they attended 
a REL-sponsored conference, training, or workshop; 44 percent said they attended a 
meeting at which a REL representative was present; 38 percent said that they or their 
organization contacted the REL for research or other assistance; 27 percent contacted a 
reference desk for help or used the “Ask a REL” link on the REL’s website; 15 percent 
forwarded a request they had received to the REL; and 18 percent had other types of 
contact with the REL (Table 13-8). 
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Table 13-8. Percentage of administrators who had used REL services and reported having 
various types of contact with the REL serving their state in the past year— West: 
School year 2011-12 


Contact 

State administrators 

District administrators 

n 

% 

n 

% 

Contacted a reference desk or used the Ask a REL link 
on the website 

0 

0 

32 

27 

Attended a REL-sponsored conference, training, or 
workshop 

9 

52 

58 

48 

A REL representative was present at a meeting or 
workshop 

14 

82 

54 

44 

Contacted REL for research or other assistance 

16 

93 

45 

38 

Forwarded a request to the REL 

3 

17 

18 

15 

Other type of contact 

0 

0 

22 

18 


NOTE: The total N state administrators was 17, and the total N for district administrators was 126. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 


How satisfied with the REL program were state and district administrators? 


Sixty-one percent of state administrators and 34 percent of district administrators in the 
West region who were atleast ^ a little familiar” with the REL program overall were “very 
satisfied” with it. 


■ Of the state administrators in the West region who were at least “a little familiar” with 
the REL program overall, 39 percent reported being “somewhat satisfied” with it and 0 
percent of state administrators reported being “not at all satisfied” with it (Table 13-9). 

■ Of the district administrators in the West region who were at least “a little familiar” with 
the REL program overall, 53 percent reported being “somewhat satisfied” with it and 
14 percent reported being “not at all satisfied” with it (Table 13-9). 

Table 13-9. Percentage of the region’s administrators at least “a little familiar” with the REL 
program who were “very satisfied,” “somewhat satisfied,” or “not at all satisfied” 
with it— West: School year 2011-12 


Satisfaction 

State administrators 

District administrators 

n 

% 

n 

% 

Very satisfied 

10 

61 

65 

34 

Somewhat satisfied 

6 

39 

102 

53 

Not at all satisfied 

0 

0 

27 

14 


NOTE: Percentages may not sum to 100 due to rounding. 


SOURCE: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratories (REL) Customer Survey, fall 2011 to spring 2012. 
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Appendix A 

Rubrics for Expert Panel Review of Impact Studies 


RUBRIC AND SCORING FORM FOR THE QUALITY AND RELEVANCE OF 
REGIONAL EDUCATIONAL LABORATORIES’ IMPACT STUDIES (FOR 

PROPOSALS) 


REL: 

Pro) ect: 

Reviewer and Date: 

SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: QUALITY OF THE PROJECT: Synthesis rating (mean of 
1A through 1J) 


Sub-dimension 1: Intervention 


Indicator 1A. The intervention for the treatment group and the condition for the 
control group are clearly described. 


Indicator IB. The plan to document implementation of the intervention (e.g., 
exposure, quality of delivery, adherence) is appropriate. 


Sub-dimension 2: Study design 


Indicator 1C. The research questions are explicitly stated, aligned with the study, and 
are empirically testable. 


Indicator ID. The design for the random assignment is rigorous. 


Indicator IE. The sampling strategies are appropriate for targeted populations, and 
the resulting sample size(s) for the impact questions will have adequate 
statistical power. 


Indicator IF. The data collection plan is appropriate for the research questions. 


Indicator 1G. Outcome measures are valid, reliable, and not overly aligned with the 
intervention. 


Sub-dimension 3: Study implementation 


Indicator 1H. Data collection procedures are clearly described. 


Indicator 11. The statistical analyses will use appropriate methods for the research 
questions (e.g., adjusting for confounding factors, clustering, missing 
data from nonresponse, attrition, multiple comparisons, and/or 
baseline non-equivalence between groups, when necessary). 


Indicator 1J. The proposed research is feasible to be carried out with reasonable 
timeline and sequencing of activities. 
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Appendix A 

Rubrics for Expert Panel Review of Impact Studies 


Dimensions and Indicators 

Rating 

DIMENSION 2: RELEVANCE OF THE PROJECT: Synthesis rating (mean 
of 2A through 2E) 


Indicator 2A. The proposal focuses on a salient topic of interest that will inform 

decisions about policies, programs, or practices in the region served by 
the REL. 


Indicator 2B. The proposal provides a thorough summary of key literature and/ or 
previous research in the topic area. 


Indicator 2C. The proposal provides a strong justification for selecting the particular 
intervention that is being studied. 


Indicator 2D. The report will contribute new information about the effectiveness of 
the intervention being studied and the more general topic being 
addressed. 


Indicator 2E. The proposal is clearly-written and well-presented. 


GENERAL COMMENTS 
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SPECIFIC RATING SHEET 


Dimension 1: Quality ( Circle the number that best describes each indicator). 



Very strong 

Strong 

Adequate 

Weak 

Very weak 

Indicator 

5 

4 

3 

2 

1 

1A. The intervention for the 
treatment group and the 
condition for the control 
group are clearly 
described. 

The proposal 
clearly describes 
both the 
intervention for 
the treatment 
group and the 
condition for the 
control group (e.g., 
business as usual, 
alternative 
treatment, no 
treatment). 


The proposal 
describes the 
intervention for 
the treatment 
group but is not 
sufficiently explicit 
about what the 
control group 
receives, or vice 
versa. 


The proposal does 
not clearly describe 
either the 
intervention for 
the treatment 
group or the 
condition for the 
control group. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

IB. The plan to document 
implementation of the 
intervention (e.g., 
exposure, quality of 
delivery, adherence) is 
appropriate. 

The plan to 
document 
implementation is 
appropriately 
detailed and 
aligned with the 
overall study 
design. 


There is a plan to 
document 
implementation 
but it should be 
more detailed 
and/ or better 
aligned with the 
study design. 


The proposal does 
not address 
implementation or 
the plan is 
seriously flawed. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1C. The research questions 
are explicitly stated, 
aligned with the study, 
and are empirically 
testable. 

The research 
questions are 
clearly articulated, 
aligned with the 
study, and can be 
empirically tested. 


Some but not all 
research questions 
are stated clearly, 
aligned with the 
study, and can be 
empirically tested. 


The proposal does 
not explicidy state 
any research 
questions; or stated 
questions are 
poorly aligned with 
the study and/ or 
cannot be 
empirically tested. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

ID. The design for the 
random assignment is 
rigorous and sound. 

The random 
assignment 
procedure 
represents the 
most rigorous 
approach to yield 
the impact 
evidence in the 
study context. 


Although a 
random 
assignment 
procedure is used, 
a better design or 
features can be 
used in the study 
context. 


The random 
assignment design 
is seriously flawed. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 



Very strong 

Strong 

Adequate 

Weak 

Very weak 

Indicator 

5 

4 

3 

2 

1 

IE. The sampling strategies 
are appropriate for 
targeted populations, and 
the resulting sample 
size(s) for the impact 
questions will have 
adequate statistical 
power. 

The sampling 
strategies are most 
appropriate for 
targeted 
populations in 
addressing the 
research questions. 
The resulting 
sample size(s) for 
the impact 
questions will have 
adequate statistical 
power. 


The sampling 
strategies are 
reasonable for 
targeted 
populations in 
addressing the 
research questions, 
although there may 
be some potential 
sampling bias. The 
sample size(s) for 
the impact 
questions will have 
adequate statistical 
power. 


The sample for 
targeted 

populations may 
contain serious 
sampling bias, 
and/ or sample 
size(s) for the 
impact questions 
will not have 
enough statistical 
power. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

IF. The data collection plan 
is appropriate for the 
research questions. 

All data sources, 
mode, timing, and 
frequency of 
collection are 
clearly defined and 
appropriate for 
addressing the 
research questions. 


Data sources, 
mode, timing, and 
frequency of 
collection are 
adequate for 
addressing the 
research questions, 
but some could be 
improved. 


Data sources, 
mode, timing, and 
frequency of 
collection are 
either not 
identified explicitly 
or not well suited 
for addressing the 
research questions. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1G. Outcome measures are 
valid, reliable 1 and not 
overly aligned with the 
intervention. 

All of the outcome 
measures are valid, 
reliable and not 
overly aligned for 
addressing the 
research questions. 


The alignment, 
validity and/ or 
reliability of some 
outcome measures 
are in question. 


The outcome 
measures have 
little if any validity 
or reliability for 
addressing the 
research questions 
or are overly 
aligned with the 
intervention, or the 
proposal does not 
address this. 


Comments on rating 


1 The WWC standards for test-retest reliability are 0.40 or higher; internal consistency is 0.60 or higher; inter-rater reliability is 0.50 or higher, where applicable. 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1H. Data collection 

procedures are clearly 
described. 

Data collection 
procedures are 
clearly described. 


Data collection 
procedures are 
described but 
could be clearer. 


Data collection 
procedures are not 
described. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

11. The statistical analyses 
will use appropriate 
methods for the research 
questions (e.g., adjusting 
for confounding factors, 
clustering, missing data, 
from nonresponse, 
attrition, multiple 
comparisons, and/ or 
baseline non-equivalence 
between groups, when 
necessary). 

The statistical 
analyses will use 
the most 
appropriate 
methods for the 
research questions 
to adjust for 
confounding 
factors, clustering, 
missing data from 
nonresponse, 
attrition, multiple 
comparisons, 
and/ or baseline 
non-equivalence 
between groups, 
when necessary. 


The statistical 
analysis methods 
are reasonable for 
the research 
questions in 
adjusting for 
confounding 
factors, clustering, 
missing data from 
nonresponse, 
attrition, multiple 
comparisons, 
and/ or baseline 
non-equivalence 
between groups, 
although better 
ones could be 
applied within the 
context, when 
necessary. 


The study 
proposes few if 
any appropriate 
statistical methods 
for the research 
questions to adjust 
for confounding 
factors, clustering, 
missing data from 
nonresponse, 
attrition, multiple 
comparisons, 
and / or baseline 
non-equivalence 
between groups, 
when necessary. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1J. The proposed research is 
feasible to be carried out 
with reasonable timeline 
and sequencing of 
activities. 

The timeline and 
sequencing of 
activities are very 
feasible and 
reasonable. 


The timeline and 
sequencing of 
activities are 
mostly feasible and 
reasonable, but 
can be improved. 


The timeline and 
sequencing of 
activities are not 
feasible or 
reasonable. 


Comments on rating 
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Dimension 2: Relevance ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2A. The proposal focuses on 
a salient topic of interest 
that can be used to 
inform decisions about 
policies, programs, or 
practices in the region 
served by the REL. 

The proposal 
provides strong 
justifications on 
why the topic was 
selected. 


The proposal 
provides some 
justifications on 
why the topic was 
selected but could 
have made a 
stronger case for 
its importance. 


The proposal 
provides little 
justifications if any 
on why the topic 
was selected or 
why it’s important. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2B. The proposal provides a 
thorough summary of key 
literature and / or previous 
research in the topic area. 

The proposal 
provides a 
thorough literature 
review of relevant 
and important 
research. 


The literature 
review is adequate 
but may overlook 
some important 
work or include 
studies that are not 
well conducted, 
without 

consideration of 
their limitations. 


The proposal does 
not include any 
review of relevant 
literature. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2C. The proposal provides a 
strong justification for 
selecting the particular 
intervention that is being 
studied. 

The proposal 
provides strong 
justification for 
selecting the 
particular 
intervention that is 
being studied. 


The proposal 
provides some 
justification for 
selecting the 
particular 
intervention that is 
being studied. 


The proposal 
provides little 
justification for 
selecting the 
particular 
intervention that is 
being studied. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2D. The proposed study will 
contribute new information 
about the effectiveness of the 
intervention and the more 
general topic being addressed. 

The proposed 
study will be 
current, new, or 
make a 

contribution to 
the intended 
audience in the 
region. 


The proposed 
study will be 
somewhat limited 
in providing 
current or new 
information, or in 
making a 
contribution to 
the intended 
audience in the 
region. 


The proposed 
study will not 
provide any 
current or new 
information or 
will not make a 
contribution to 
the intended 
audience in the 
region. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2E. The proposal is clearly 
written and well- 
presented. 

The proposal 
contains clear, 
comprehensive, 
and accurate 
information about 
the design and 
conduct of the 
research. 


The proposal 
describes the 
proposed design 
and conduct of the 
research but could 
be more clear. 


The proposal is 
poorly written, 
with critical 
information 
missing about the 
design and 
conduct of the 
research. 


Comments on rating 
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Rubrics for Expert Panel Review of Impact Studies 


RUBRIC AND SCORING FORM FOR THE QUALITY AND RELEVANCE OF 
REGIONAL EDUCATIONAL LABORATORIES’ IMPACT STUDIES (FOR FINAL 

REPORTS) 


REL: 

Pro) ect: 

Reviewer and Date: 

SUMMARY SHEET 


Dimensions and Indicators 

Rating 

DIMENSION 1: QUALITY OF THE PROJECT: Synthesis rating (mean of 1A 
through lO) 


Sub-dimension 1: Intervention 


Indicator 1A. The intervention for the treatment group and the condition for the 
control group are clearly described. 


Indicator IB. Implementation of the intervention is well documented (e.g., exposure, 
quality of delivery, adherence). 


Indicator 1C. There is minimal contamination in the form of crossover between 
subjects in treatment and control condition or spillover of the 
intervention from the treatment to the control group. 


Sub-dimension 2: Study design and implementation 


Indicator ID. The research questions are explicitly stated, aligned with the study, and 
are empirically testable. 


Indicator IE. The design and implementation of random assignment are rigorous. 


Indicator IF. The sampling strategies are appropriate for targeted populations and the 
resulting sample size(s) for the impact questions have adequate statistical 
power. 


Indicator 1 G. Outcome measures are valid and reliable and not overly aligned with the 
intervention. 


Indicator 1H. The data collection plan is appropriate for the research questions. 


Indicator 11. The data collection plan is well-implemented. 


Indicator 1J. The overall attrition rate and differential attrition rates are acceptable 
given the length of the intervention. 


Indicator IK. The data analyses use appropriate methods to address the research 

questions (e.g., adjusting where necessary for any clustering, missing data 
from nonresponse, multiple comparisons, and/ or baseline non- 
equivalence between groups). 


Sub-dimension 3: Reporting 


Indicator 1L. Appropriate statistics are provided to describe the sample and support 
the findings. 


Indicator 1M. The conclusions about the intervention are drawn appropriately and 
consistently. 


Indicator IN. All of the research questions are specifically addressed by the analyses. 


Indicator lO. The limitations of the study are clearly and comprehensively stated. 
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Dimensions and Indicators 

Rating 

DIMENSION 2: RELEVANCE OF THE PROJECT: Synthesis rating (mean 
of 2A through 2F) 


Indicator 2A. The report provides a strong justification why the topic was selected as 
salient to meet the needs of the region served by the REL. 


Indicator 2B2B. The report provides a thorough summary of key literature 

and/ or previous research in the topic area. 


Indicator 2CC. The report provides a strong justification for selecting the particular 
intervention that is being studied. 


Indicator 2DD. The report contributes new information about the effectiveness 

of the intervention being studied and the more general topic being 
addressed. 


Indicator 2E. The executive summary of the report is easy to read and understand for 
a lay audience. 


Indicator 2F. The report is clear and well-written for the technical audience. 
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SPECIFIC RATING SHEET 


Dimension 1: Quality ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1A. The intervention for the 
treatment group and the 
condition for the control 
group are clearly described. 

The report clearly 
describes both 
the intervention 
for the treatment 
group and the 
condition for the 
control group 
(e.g., business as 
usual, alternative 
treatment, no 
treatment) 


The report 
describes the 
intervention for 
the treatment 
group but is not 
sufficiently 
explicit about 
what the control 
group receives, or 
vice versa. 


The report does 
not clearly 
describe either 
the intervention 
for the treatment 
group or the 
condition for the 
control group. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

IB. Implementation of the 
intervention is well 
documented (e.g., exposure, 
quality of delivery, 
adherence). 

Implementation 
of the 

intervention is 
well-documented. 


Implementation 
of the 

intervention is 
documented but 
could be more 
detailed. 


Implementation 
of the 

intervention is 
poorly 

documented or 
not addressed in 
the report. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1C. There is minimal 

contamination in the form of 
crossover between subjects in 
treatment and control 
condition or spillover of the 
intervention from the 
treatment to the control 
group. 

Contamination in 
the form of 
crossover or 
spillover is 
minimal. 


Contamination in 
the form of 
crossover or 
spillover is 
present but at 
modest levels. 


Contamination in 
the form of 
crossover or 
spillover is 
serious. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

ID. All research questions are 
explicitly stated, aligned with 
the study, and are empirically 
testable. 

All research 
questions are 
clearly articulated; 
aligned with the 
study, and can be 
empirically tested. 


Some, but not all, 
research 
questions are 
stated clearly, 
aligned with the 
study, and can be 
empirically tested. 


The report does 
not explicidy state 
any research 
questions, or 
stated questions 
are poorly aligned 
with the study 
and/ or cannot be 
empirically tested. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

IE. The design and 

implementation of random 
assignment are rigorous. 

The random 
assignment design 
and 

implementation 
represents the 
most rigorous 
approach to yield 
the impact 
evidence in the 
study context. 


Although a 
random 
assignment 
procedure is used, 
a better design or 
implementation 
features could 
have been used in 
the study context. 


The random 
assignment design 
or 

implementation is 
seriously flawed. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 



Very strong 

Strong 

Adequate 

Weak 

Very weak 

Indicator 

5 

4 

3 

2 

1 

IF. The sampling strategies are 
appropriate for targeted 
populations, and the resulting 
sample size(s) for the impact 
questions have adequate 
statistical power. 

The sampling 
strategies are 
most appropriate 
for targeted 
populations and 
addressing the 
research 
questions. The 
resulting sample 
size(s) for the 
impact questions 
have adequate 
statistical power. 


The sampling 
strategies are 
reasonable for 
targeted 
populations in 
addressing the 
research 
questions, 
although there 
may be some 
potential 
sampling bias. 
The sample 
size(s) for the 
impact questions 
are not 
sufficiently 
powered. 


The sample for 
targeted 

populations may 
contain serious 
sampling bias, 
and/ or sample 
size(s) for the 
impact questions 
do not have 
enough statistical 
power. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1 G. Outcome measures are valid, 
reliable, 1 and not overly 
aligned with the intervention. 

All of the 
outcome 
measures are 
valid, reliable and 
not overly aligned 
for addressing the 
research 
questions. 


The alignment, 
validity and/ or 
reliability of some 
outcome 
measures are in 
question. 


The outcome 
measures have 
little if any validity 
or reliability for 
addressing the 
research 
questions or are 
overly aligned 
with the 
intervention, or 
the report does 
not address this 
aspect. 


Comments on rating 


1 The WWC standards for test-retest reliability are 0.40 or higher; internal consistency is 0.60 or higher; inter-rater reliability is 0.50 or higher, where applicable. 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1H. The data collection plan is 
appropriate for the research 
questions. 

All data sources 
are clearly defined 
and mode, timing, 
frequency of 
collection are 
appropriate for 
addressing the 
research 
questions. 


Data sources, 
mode, timing, and 
frequency of 
collection are 
adequate for 
addressing the 
research 
questions, but 
some could be 
improved. 


Data sources, 
mode, timing, and 
frequency of 
collection are 
either not 
identified 
explicitly or not 
well suited for 
addressing the 
research 
questions. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1 H. Outcome measures are valid, 
reliable,” and not overly 
aligned with the intervention. 

All of the 
outcome 
measures are 
valid, reliable and 
not overly aligned 
for addressing the 
research 
questions. 


The alignment, 
validity and/ or 
reliability of some 
outcome 
measures are in 
question. 


The outcome 
measures have 
little if any validity 
or reliability for 
addressing the 
research questions 
or are overly 
aligned with the 
intervention, or 
the report does 
not address this 
aspect. 


Comments on rating 


2 The WWC standards for test-retest reliability are 0.40 or higher; internal consistency is 0.60 or higher; inter-rater reliability is 0.50 or higher, where applicable. 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1J. The data collection plan is 
well implemented. 

The data 
collection plan is 
implemented 
essentially as 
designed. 


There are some 
concerns with the 
implementation 
of the data 
collection plan. 


There are serious 
concerns with the 
implementation 
of the data 
collection plan. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 



Very strong 

Strong 

Adequate 

Weak 

Very weak 

Indicator 

5 

4 

3 

2 

1 

1J. The overall attrition rate and 

The overall 


The overall 


The overall 

differential attrition rates are 

attrition rate and 


attrition and/ or 


attrition or 

acceptable given the length of 

differential 


differential 


differential 

the intervention. 

attrition rates 


attrition rates are 


attritions are very 


between 


slightly high. 


high. The source 


treatment and 


However, the 


of attrition is 


control groups 


source of attrition 


mostly 


are low given the 


is mostly 


endogenous. 


length of the 


exogenous (e.g., 




intervention. 


parent mobility 
with young 
children) rather 
than endogenous 
(e.g., students 
choosing whether 
to participate in 
an intervention). 



Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator). 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1J. Data collection plan is well- 
implemented. 

The data 
collection plan is 
implemented 
essentially as 
designed. 


There are some 
concerns with 
how data were 
collected. 


There are serious 
concerns with the 
data collection 
procedures. 


Comments on rating 


Appendix A 

Rubrics for Expert Panel Review of Impact Studies 


Evaluation of the Regional Educational Laboratories: Final Report 

A-32 


Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 



Very strong 

Strong 

Adequate 

Weak 

Very weak 

Indicator 

5 

4 

3 

2 

1 

IK. The data analyses use 
appropriate methods to 
address the research questions 
(e.g., adjusting where 
necessary for any clustering, 
missing data from 
nonresponse, multiple 
comparisons, and/ or baseline 
non-equivalence between 
groups). 

The data analyses 
use the most 
appropriate 
methods to 
address the 
research 
questions (e.g., 
adjusting where 
necessary for 
clustering, 
missing data from 
nonresponse, 
multiple 
comparisons, 
and/ or baseline 
non-equivalence 
between groups). 


The data analyses 
use reasonable 
methods to 
address the 
research 
questions 
although better 
methods could be 
applied within the 
context. 


The data analyses 
use few, if any, 
appropriate 
methods to 
address the 
research 
questions. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1L. Appropriate statistics are 
provided to describe the 
sample and support the 
findings. 

Appropriate 
descriptive and 
inferential 
statistics, 
respectively, are 
provided to 
describe the 
sample and 
support the 
findings about the 
impact. 


Some, but not all, 
appropriate 
descriptive and 
inferential 
statistics, 
respectively, are 
provided to 
describe the 
sample or 
support the 
findings about the 
impact. 


The report does 
not provide 
descriptive 
statistics about 
the sample 
and/ or inferential 
statistics about 
the impact. 


Comments on rating 


Appendix A 

Rubrics for Expert Panel Review of Impact Studies 


Evaluation of the Regional Educational Laboratories: Final Report 

A-34 


Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

1M. The conclusions about the 
intervention are drawn 
appropriately and consistently. 

All of the 
intervention are 
adequately 
addressed by the 
report. 


Some conclusions 
are either over- 
stated or under- 
stated, or the 
conclusions are 
stated slightly 
inconsistently 
throughout the 
report. 


None of the 
research 
questions is 
adequately 
addressed by the 
analyses. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

IN. All of the research questions 
are specifically addressed by 
the analyses. 

All of the 
research 
questions are 
specifically 
addressed by the 
analyses. 


Some, but not all, 
of the research 
questions are 
specifically 
addressed by the 
analyses. 


None of the 
research 
questions is 
specifically 
addressed by the 
analyses. 


Comments on rating 
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Dimension 1: Quality, continued ( Circle the number that best describes each indicator) 


Indicator 

Very strong 

Strong 

Adequate 

Weak 

Very weak 

5 

4 

3 

2 

1 

lO. The limitations of the study 
are clearly and 
comprehensively stated. 

The limitations of 
the study are 
described clearly 
and 

comprehensively. 


The limitations of 
the study are 
acknowledged but 
may not be clear 
or 

comprehensive. 


The report 
includes little if 
any information 
on the limitations 
of the study. 


Comments on rating 
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Dimension 2: Relevance ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2A. The report provides a strong 
justification why the topic was 
selected or focuses on a salient 
topic of interest that will 
inform decisions about 
policies, programs, or 
practices to meet the needs of 
the region served by the REL. 

The report 
provides strong 
justifications on 
why the topic was 
selected. . 


The report 
provides some 
justification on 
why the topic was 
selected but could 
have made a 
stronger case for 
its importance. 


The report 
provides little 
justification on 
why the topic was 
selected or why 
it’s important. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2B. The report provides a 

thorough summary of key 
literature and / or previous 
research in the topic area. 

The study 
provides a 
thorough 
literature review 
of relevant and 
important 
research. 


The literature 
review is 

adequate, but may 
overlook some 
important work 
or include studies 
that are not well 
conducted, 
without 

consideration of 
their limitations. 


The report does 
not include any 
review of relevant 
literature. 


Comments on rating 
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Dimension 2: Relevance, continued (Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2C. The report provides a strong 
justification for selecting the 
particular intervention that is 
being studied. 

The report 
provides strong 
justifications for 
selecting the 
particular 
intervention that 
is being studied. 


The report 
provides some 
justification for 
selecting the 
particular 
intervention that 
is being studied. 


The report 
provides little 
justification for 
selecting the 
particular 
intervention that 
is being studied. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2D. The report contributes new 
information about the 
effectiveness of the 
intervention being studied and 
the more general topic being 
addressed. 

The information 
presented in the 
report is current, 
new, or makes a 
contribution to 
the intended 
audience in the 
region. 


The report is 
somewhat limited 
in providing 
current or new 
information or in 
making a 
contribution to 
the intended 
audience in the 
region. 


The report does 
not provide any 
current or new 
information or 
does not make a 
contribution to 
the intended 
audience in the 
region. 


Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2E. The executive summary of the 
report is easy to read and 
understand for a lay audience. 







Comments on rating 
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Dimension 2: Relevance, continued ( Circle the number that best describes each indicator) 


Indicator 

Highly relevant 

Relevant 

Adequate 

Marginally 

relevant 

Not relevant 

5 

4 

3 

2 

1 

2F. The report is clear and well- 
written for the technical 
audience. 

The report is 
well- written for 
the technical 
audience, 
containing clear, 
comprehensive, 
and accurate 
information 
about the design 
and conduct of 
the research. 


The report is 
moderately well- 
written for the 
technical 
audience, but 
could be more 
clear or 

comprehensive 
about the design 
and conduct of 
the research. 


The report is 
poorly written for 
the technical 
audience, with 
critical 
information 
missing about the 
design and 
conduct of the 
research. 


Comments on rating 
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Appendix B 
Inter-rater Agreement 


When protocols require multiple sets of raters to assign ratings to targets (in this case, proposals and 
final reports), it is important to assess two related but conceptually distinct concepts: inter-rater 
agreement and inter-rater reliability. Levels of inter-rater agreement (IRA) index the extent to which 
raters assign the same scores to common targets, while inter-rater reliability (IRR) indicates the 
extent to which raters provide consistency in relative judgments. For example, if one rater provides 
ratings that are exactly 2 points higher than another rater, measures of inter-rater reliability will be 
high, while inter-rater agreement will be lower. For the expert panel review of REL products, IRA 
was identified as more conceptually appropriate for assessing the extent to which raters provided 
similar ratings. 

There are a wide variety of methods for assessing IRA and IRR, including measures such as Cohen’s 
kappa and various intraclass correlation coefficients (ICC). However, these measures were less than 
optimal for purposes of the current study. For example, Cohen’s kappa is a classic measure of 
agreement, but requires additional modifications (e.g., weighting schemes) to handle ranked data and 
is primarily designed around assessing agreement dyadically — e.g., between pairs of individual raters 
on individual items. The various ICC coefficients (c.f, Shrout and Fleiss, 1979) are widely used to 
assess the consistency of ratings, but provide measures based on both IRA and IRR, rather than 
IRA alone (LeBreton and Senter, 2008), do not provide a measure of agreement for individual 
products, and are based on mean dimension-level ratings, rather than item-level responses. 

For the expert panel review of REL proposals and IES-published reports, IRA was indexed using 
the ^(d measure (James, Demaree, and Wolf, 1984). This measure is widely used in the 
psychometric, management, and psychology literatures; is applicable to multi-item ordinal response 
scales; and provides a measure of IRA for each proposal and final report being rated (as opposed to 
a single measure of IRA for the entire study) based on a definite number of raters. R« is given by 
the following formula: 


Rwg(j) 
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where J — the number of items in the rating scale, s % is the mean of the observed variances for the J 
observed items, and erj is the expected variance when there is a complete lack of agreement among 
raters (e.g., where raters respond randomly to items). For the expert panel review of REL proposals 
and IES-published reports, crj =2, based on the expected variance of a uniform response 
distribution for a 5-item response scale. R~ ranges from 0 to 1, with 0 indicating a complete lack of 
agreement and 1 indicating perfect agreement. 

The analysis was conducted using the rwg.j function from the multilevel package in R version 2.10.1 
(R Foundation for Statistical Computing 2010); codes are also provided for SPSS in LeBreton and 
Senter (2008). Because the Rwg(j) formula expects numeric ratings, “NA” ratings were treated as 
missing data for purposes of the Rwg(j) analysis. 1 Missing data are known to have an effect on R^ (j) 
estimates (Newman and Sin 2009), such that responses missing not at random (MNAR) tend to 
produce overestimates of R with the level of bias dependent on the level of missingness in the 
data and the magnitude of the relationship between missingness and the variable of interest. 

As with many metrics of IRA, what constitutes a “high” or “low” level of agreement will ultimately 
depend on the purposes of the ratings, but a review by LeBreton and Senter (2008) suggested the 
following interpretations of R q: ranges from 0.00 to 0.30 represent a lack of agreement; 0.31 to 
0.50 represents weak agreement; 0.51 to 0.70 is moderate agreement; 0.71 to 0.90 is strong 
agreement; and 0.91 to 1.00 is very strong agreement. In order to identify products for further 
reconciliation and re-review by raters, products were selected for reconciliation if they had an R wg(|j 
of 0.30 or less on either quality or relevance (the two dimensions). 

In addition, for each product thus identified, raters were provided with a list of specific indicators 
for which the greatest level of disagreement existed, defined by the inter-rater variance for specific 
indicators. These were identified by selecting indicators for which ratings had a variance of two or 
more; this threshold was selected based on the expected variance of a uniform distribution of an 
ordinal scale with five response categories, such that indicators with variances less than two tended 
to have greater levels of agreement than might be expected based on chance responding. Flowever, 
note that this threshold does not represent a statistical test of inter-rater agreement; rather, it was a 
heuristic designed to help raters quickly identify and resolve the largest discrepancies in their ratings 
in the limited time available. 


1 The base rwg.j function provided by the multilevel package had to be modified slightly to handle missing data; this is 
due to the default behavior of certain R functions for the mean and variance, rather than due to issues with the 
algorithm implemented by the rwg.j function or the Rwg(j) formula itself. 
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Appendix C 

Distribution of Indicator-Level Expert Panelist Ratings 


Table C-l. Distribution of indicator-level expert panelist ratings (on a 5-point scale, with 5 being 
the highest) for REL impact study proposals and lES-published reports on quality 
and relevance 


Quality rating 

Percent of impact study 
proposals (%) 

Percent of lES-published impact study 

reports (%) 

Very weak 

2.0 

0.3 

Weak 

13.1 

4.4 

Adequate 

29.3 

21.1 

Strong 

35.0 

33.1 

Very strong 

20.5 

41.1 

Relevance rating 



Not relevant 

0.6 

0.0 

Marginally relevant 

9.7 

0.7 

Adequate 

33.3 

24.3 

Relevant 

41.2 

43.1 

Highly relevant 

15.2 

31.9 


NOTE: Proposals were submitted by the RELs between March 1, 2006, and September 1, 2011; reports were published by IES between 
November 2009 and April 2011. The distributions for proposal quality and relevance were based on 297 and 165 indicator-level 
ratings, respectively. The distributions for report quality and relevance were based on 360 and 144 indicator-level ratings, respectively. 
Differences are largely attributed to the number of indicators in each dimension. Percentages may not sum to 100 due to rounding. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports 
prepared under the 2006-2011 REL contracts and published by IES by September 1, 2011); Rubric and Scoring Form for the Quality 
and Relevance of Regional Educational Laboratories’ Impact Study Projects (for proposals for impact studies under the 2006-2011 
REL contracts submitted by September 1, 2011) (Appendix A). 
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Table C-2. Distribution of indicator-level expert panelist ratings (on a 5-point scale, with 5 being the highest) for REL impact study 
proposals and lES-published reports on quality and relevance, by REL 


Impact study proposals 

AP 

CE 

MA 

MW 

NE&I 

NW 

PA 

SE 

SW 

W 

Quality rating 











Very weak 

0.0 

0.0 

0.0 

11.1 

0.0 

0.0 

3.7 

0.0 

0.0 

3.7 

Weak 

14.8 

7.4 

0.0 

25.9 

14.8 

18.5 

40.7 

3.7 

0.0 

9.3 

Adequate 

29.6 

18.5 

18.5 

25.9 

22.2 

63.0 

33.3 

25.9 

29.6 

27.8 

Strong 

40.7 

44.4 

40.7 

37.0 

48.1 

18.5 

11.1 

40.7 

37.0 

33.3 

Very strong 

14.8 

29.6 

40.7 

0.0 

14.8 

0.0 

11.1 

29.6 

33.3 

25.9 

Relevance rating 











Not relevant 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Marginally relevant 

26.7 

6.7 

13.3 

0.0 

0.0 

20.0 

26.7 

0.0 

0.0 

6.7 

Adequate 

46.7 

40.0 

26.7 

60.0 

46.7 

26.7 

33.3 

6.7 

13.3 

26.7 

Relevant 

20.0 

26.7 

53.3 

33.3 

46.7 

40.0 

26.7 

35.6 

66.7 

40.0 

Highly relevant 

0.0 

26.7 

6.7 

6.7 

6.7 

13.3 

13.3 

57.8 

20.0 

26.7 

lES-published reports 











Quality rating 











Very weak 

— 

0.0 

2.2 

— 

0.0 

0.0 

— 

0.0 

0.0 

0.0 

Weak 

— 

6.7 

0.0 

— 

0.0 

15.6 

— 

0.0 

0.0 

6.7 

Adequate 

— 

15.6 

8.9 

— 

22.2 

26.7 

— 

20.0 

42.2 

23.3 

Strong 

— 

37.8 

31.1 

— 

37.8 

35.6 

— 

60.0 

28.9 

28.9 

Very strong 

— 

40.0 

57.8 

— 

40.0 

22.2 

— 

20.0 

28.9 

41.1 

Relevance rating 











Not relevant 

— 

0.0 

0.0 

— 

0.0 

0.0 

— 

0.0 

0.0 

0.0 

Marginally relevant 

— 

5.6 

0.0 

— 

0.0 

0.0 

— 

0.0 

0.0 

0.0 

Adequate 

— 

38.9 

5.6 

— 

38.9 

44.4 

— 

5.6 

27.8 

16.7 

Relevant 

— 

33.3 

50.0 

— 

55.6 

44.4 

— 

33.3 

44.4 

41.7 

Highly relevant 

— 

22.2 

44.4 

— 

5.6 

11.1 

— 

61.1 

27.8 

41.7 


—Not applicable 


NOTE: Proposals were submitted by the RELs between March 1, 2006, and September 1, 2011; reports were published by IES between November 2009 and April 2011. Percentages may 
not sum to 100 due to rounding. 

SOURCE: Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories’ Impact Study Projects (for reports prepared under the 2006-2011 REL contracts and 
published by IES by September 1, 2011); Rubric and Scoring Form for the Quality and Relevance of Regional Educational Laboratories' Impact Study Projects (for proposals for impact 
studies under the 2006-2011 REL contracts submitted by September 1, 2011) (Appendix A). 
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Summary of Impact Studies Reviewed 


Review of the Report "Classroom Assessment for Student Learning: 
The Impact on Elementary School Mathematics in the Central 
Region” 

Citation: Randel, B., Beesley, A.D., Apthorp, H., Clark, T.F., Wang, X., Cicchinelli, L.F., & 

Williams, J.M. (201 1 ) . Classroom Assessment for Student learning: The impact on elementary school mathematics 
in the Central Region. (NCEE 2011-4005). Washington, DC: National Center for Education 
Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 

What is this study about? 

The study used a random assignment design to investigate the impact of Classroom Assessment for 
Student Teaming (CAST) on elementary students’ mathematics achievement. 

A total of 67 schools across 32 Colorado school districts were randomly assigned to either an 
intervention condition that used CAST or a comparison condition that did not use CAST. The 
study analyzed data from 2,860 students in 33 schools with CAST and 3,379 students in 34 
comparison schools without CAST. Fourth- and fifth-grade teachers in the intervention schools 
studied the CAST materials and applied CAST principles, practices, and tools in their classrooms 
during the training year. The intervention teachers then implemented the CAST program in their 
classrooms for one full school year. Teachers in the comparison group took part in their regular 
professional development activities. 

The study assessed the effectiveness of the CAST program by comparing mathematics achievement 
of students in the C4ST and comparison groups in the spring of the implementation year. 


Features of CASL 

CAST is a professional development program on classroom and formative assessment published by 
the Assessment Training Institute of Pearson Education. The CAST program includes a textbook, 
DVDs, ancillary books, and an implementation handbook, all of which are used to train teachers to 
conduct classroom assessments that are appropriate for and aligned with their learning targets. 
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CASL is typically implemented via teacher learning teams, in which teachers meet regularly to 
discuss and reflect on the content of the textbooks and DVDs and to share their experiences 
applying the program in their classrooms. Part of CL4SL’s approach is to increase student 
involvement in all aspects of assessment. 

This study hypothesized that use of CASL would increase teachers’ knowledge and quality of 
classroom assessment practices, which in turn would lead to improved student motivation and math 
achievement. 

What did the study find? 

The study found no effects of CASL on the mathematics achievement of fourth- and fifth-grade 
students. The estimated effect size of 0.01 is neither statistically significant nor substantively 
important. 


Review of the Report “A Multisite Cluster Randomized Trial of the 
Effects of CompassLearning Odyssey® Math on the Math 
Achievement of Selected Grade 4 Students in the Mid-Atlantic 
Region” 

Citation: Wijekumar, K., Hitchcock, J., Turner, H., Lei, PW., & Peck, K. (2009). A multisite cluster 
randomised trial of the effects of CompassLearning Odyssey® Math on the math achievement of selected grade 4 
students in the Mid-Atlantic Legion (NCEE 2009-4068). Washington, DC: National Center for 
Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 
Education. 

What is this study about? 

The study examined whether exposure to Odyssey® Math , a web-based mathematics curriculum and 
assessment tool, improved mathematics achievement of fourth-grade students. The study analyzed 
data from 2,456 fourth-grade students in 122 classrooms in 32 elementary schools in Delaware, New 
Jersey, and Pennsylvania. 

The study randomly assigned all fourth-grade classrooms in 32 elementary schools to intervention or 
comparison conditions. Intervention classrooms used Odyssey® Math for 60 minutes each week 
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during the entire school year as a partial substitute for the regular curriculum; comparison 
classrooms used their school’s standard mathematics curriculum for the total math instructional 
time. 

The study assessed the effectiveness of Odyssey® Math by comparing the mathematics achievement 
of students in the intervention and comparison groups in the spring of the implementation year. 


Features of Odyssey® Math 

Odyssey® Math , published by CompassLearning, is a web-based K-8 mathematics curriculum and 
assessment tool designed to enable teachers to differentiate student instruction and make data- 
driven decisions. Odyssey® Math can be used as a standalone curriculum or as a partial substitute to 
other mathematics curricula. 

Each Odyssey® Math module contains learning activities for students that include narrative 
descriptions of how to solve problems, practice tasks, quizzes, and feedback. Modules also contain 
math tools and assessments. Specific learning activities and difficulty levels can be selected by the 
software or teacher. 

What did the study find? 

The study found no discernible effects of Odyssejf® Math on mathematics achievement in the spring 
of the implementation year. The estimated effect size of 0.02 is not statistically significant or 
substantively important. 


Review of the Report "Impact of the Thinking Reader® Software 
Program on Grade 6 Reading Vocabulary, Comprehension, 
Strategies, and Motivation” 

Citation: Drummond, K., Chinen, M., Duncan, T.G., Miller, H.R., Fryer, L., Zmach, C., & Culp, K. 
(201 1). Impact of the Thinking Reader® software program on grade 6 reading vocabulary, comprehension, strategies, 
and motivation (NCEE 2010-4035). Washington, DC: National Center for Education Evaluation and 
Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 
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What is this study about? 

The study of Thinking Render® is a multisite cluster randomized controlled trial. Ninety-two 
reading/English language arts teachers from 32 elementary and middle schools were randomly 
assigned within their schools to either the Thinking Reader® condition or the comparison condition. 
The analysis sample consisted of 90 classes and 2,147 grade 6 students, with 1,156 students in the 
ThinkingReader® condition and 991 students in the comparison condition. 

Teachers in the Thinking Reader® condition supplemented their regular English language arts or 
reading instruction with one to three preselected Thinking Reader® novels that students were asked to 
read within the Thinking Reader® software program. Students in comparison group classrooms 
participated in the schools’ regular curriculum. 

The study assessed the effectiveness of Thinking Reader® by comparing the reading comprehension 
of students in the Thinking Reader® and comparison conditions at the end of the school year. 

Features of Thinking Reader® 

Thinking Reader® is a software program that aims to motivate middle school students to read and to 
make self-directed use of seven target comprehension strategies: (a) summarizing, (b) clarifying, (c) 
visualizing, (d) reflecting, (e) questioning, (f) predicting, and (g) feeling. Students listen to a novel 
while following highlighted text on a computer screen and then respond to questions about the 
story. The program applies reciprocal teaching methods through the use of animated coaches and 
peers to enhance comprehension strategies. 

The Thinking Reader® instructional routine consists of three phases. In the first phase, teachers 
introduce students to the program through activities such as modeling a strategy. During the second 
phase, the teachers observe and review students’ progress while students read a novel on the 
computer. For the third phase, teachers and students interact offline: they discuss the book, and 
then students complete an activity to demonstrate understanding. The program has five levels of 
interactive instructional support and allows students to progress to lower levels of support where 
they can independently select comprehension strategies. 

lWhat did the study find? 

The study found no statistically significant differences on the comprehension outcomes of students 
in the Thinking Reader® classes, compared with students in the comparison classes. 
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Review of the Report "An Experimental Study of the Project CRISS 
Reading Program on Grade 9 Reading Achievement in Rural High 
Schools” 

Citation: Kushman, J., Hanita, M., and Raphael, J. (201 1). An experimental study of the Project CRISS 
reading program on grade 9 reading achievement in rural high schools. (NCEE 2010-4007). Washington, DC: 
National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, 
U.S. Department of Education. 

What is this study about? 

The study examined the effectiveness of a teacher professional development called Project CRISS 
(Creating Independence Through Student-owned Strategies j on grade 9 student reading achievement. 

Smaller mral and town high schools from six western states that enrolled 250-1,000 students were 
targeted. Fifty-two schools were randomly assigned to a treatment condition and control condition. 
Schools were blocked by cohort, state, and poverty index. 

Treatment impact was assessed using the Stanford Diagnostic Reading Test (4 th ed) Comprehension 
subtest. 

Features of Project CRISS 

Through Project CRISS, high school teachers learn how to apply research-based learning principles 
and reading/writing strategies in all major subject or content areas using materials, training, and 
follow-up support provided by the developer. 

Over two school years in 2007-09, the treatment consisted of 24 hours of formal teacher training 
plus an additional 4-5 days of onsite consultation and assistance by a certified trainer. Additionally, a 
school teacher was selected to serve as a local facilitator. 
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What did the study find? 

The study did not find any statistical significant difference in student reading comprehension test 
scores between treatment and control conditions. 


Review of the Report "The Effectiveness of a Program to Accelerate 
Vocabulary Development in Kindergarten” 

Citation: Goodson, B., Wolf, A., Bell, S., Turner, H., & Finney, P.B. (20 1 0) . The effectiveness of a 
program to accelerate vocabulary development in kindergarten (VOCAB). (NCEE 2010-4014). Washington, 
DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. 

What is this study about? 

The study examined whether exposure to Kindergarten PAVEd for Success, a vocabulary instruction 
program, improved the expressive vocabulary of kindergartners. 

The study analyzed data for nearly 1,300 kindergarten students in 64 schools serving predominandy 
mral and high-poverty youth in the Mississippi Delta region and surrounding areas. 

Eligible schools were placed into three blocks based on previous participation in reading initiatives 
and then randomly assigned within blocks to either supplement their language arts curriculum with 
the Kindergarten PAVEd for Success program or not. Prior to random assignment of schools, two 
kindergarten classrooms were randomly selected from each school for participation in the study; a 
random sample of 10 students was then drawn from each classroom. The study followed this sample 
of students in each school. 

The study assessed the Kindergarten PAVEd for Success program’s effectiveness by comparing the 
expressive vocabulary and listening comprehension of students in the treatment and comparison 
groups at the end of the school year. 
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Features of the Kindergarten PAVEd for Success Program 

The Kindergarten PAVEd for Success program is a 24- week in-class supplement to a school’s core 
language arts program. It is built around three components: 

1 . Explicit Vocabulary Instruction on a large set of target words aligned with themes in 
Mississippi’s science and social studies frameworks; 

2. Interactive Book Reading, which involves teachers asking questions that promote 
comprehension and oral language skills during story-reading time; and 

3. Adult-Child Conversations, in which teachers have frequent conversations with individual 
or small groups of students to introduce or use new vocabulary. 

What did the study find? 

Kindergarten students in schools using Kindergarten PAVEd for Success as a supplement to regular 
literacy instruction performed better than kindergarten students in comparison schools. The average 
effect size of 0.12 in the reading comprehension domain was statistically significant. 

The authors reported that students who received Kindergarten PAVEd for Success instruction were 1 
month ahead in vocabulary development at the end of kindergarten, compared with students in the 
comparison group. 


Review of the Report “The Impact of Collaborative Strategic 
Reading on the Reading Comprehension of Grade 5 Students in 
Linguistically Diverse Schools” 

Citation: Hitchcock, J., Dimino, J., Kurki, A., Wilkins, C., & Gersten, R. (2010). The impact of 
collaborative strategic reading on the reading comprehension of grade 5 students in linguistically diverse schools. 
(NCEE 201 1-4001). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education. 

What is this study about? 

The study examined the effect of Collaborative Strategic Reading fCSRJ on students’ reading 
comprehension. 


Evaluation of the Regional Educational Laboratories: Final Report 

D-7 


Appendix D 

Summary of Impact Studies Reviewed 


Study participants included 74 classrooms (37 CAR, 37 control) across 26 schools and 5 large urban 
and suburban districts with a total of 1,355 students in Oklahoma and Texas districts that serve a 
large number of ELL students (25 percent or more). Within each participating school, grade 5 social 
studies classrooms were randomly assigned to either condition. 

Reading comprehension score on the Group Reading Assessment and Diagnostic Evaluation 
(GRADE) was used as the outcome measure. 

Features of the intervention 

CAR is a set of instructional strategies designed to improve the reading comprehension of students 
with diverse abilities. Teachers implement CAR at the classroom level using scaffolded instruction to 
guide students in the independent use of four comprehension strategies; students apply the strategies 
to informational text while working in small cooperative learning groups. 

The implementation period was 1 school year. The developer provided an initial 2-day training to 
teachers. Training was provided to coaches who offered four follow-up coaching sessions to 
teachers throughout the year. About 79 percent of teachers reported using CAR two or more times a 
week as instructed. However, classroom observation found that only 22 percent were using all five 
core teacher strategies defined as full procedural fidelity. 

lWhat did the study find? 

The study did not find any impact from CAR on student reading comprehension. 


Review of the Report “Effects of Problem Based Economics on High 
School Economics Instruction” 

Citation: Finkelstein, N., Hanson, T., Huang, C.-W., Hirschman, B., & Huang, M. (2010). Effects of 
Problem Based Economics on high school economics instruction. (NCEE 2010-4002). Washington, DC: 
National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, 
U.S. Department of Education. 
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What is this study about? 

The study included 128 high school economics teachers from 106 schools in Arizona and California, 
half of whom were randomly assigned to the Problem Based Economics Instruction condition and half of 
whom were randomly assigned to the comparison condition. High levels of teacher attrition 
occurred after randomization and before implementation. The analysis sample included 64 teachers, 
with 35 in the treatment condition and 29 in the comparison condition. Student attrition was low, 
and the student analytic sample was shown to be equivalent in economic literacy at baseline. 

Intervention teachers used Problem Based Economics Instruction materials as a major portion of their 
curriculum content and instructional program during the 2007-08 academic year, whereas 
comparison teachers used their schools’ standard instructional materials. 

The study assessed the effectiveness of Problem Based Economics Instruction by comparing the 
economics knowledge of students in the treatment and comparison groups at the end of their 
implementation semester. 


Features of Problem Based Economics Instruction 

Building on the more general techniques of problem-based learning, the Problem Based Economics 
Instruction curriculum was designed by the Buck Institute for Education, with input from university 
economists and expert teachers. The intervention is intended to help students actively learn critical 
thinking and problem-solving skills using real-world examples. 

Each economics unit took place over 4 to 15 instructional days. Teachers were asked to provide core 
course content and use a set of strategies to help students contextualize, comprehend, and solve real 
economic problems; work in a group; communicate effectively using multiple methods and 
technologies; gather information and analyze data; understand interrelationships across economics 
systems; and make inferences. 

The curriculum was designed to include nine modules. Five of the nine available modules were 
selected for use in this study and were provided to the intervention group teachers; these modules 
were chosen because they included fundamental components of the curriculum standards in Arizona 
and California. 
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What did the study find? 

The study found no statistically significant difference on the economics knowledge of students in 
grades 1 1 and 12 in the classes that used Problem Based Economics Instruction , relative to students in the 
comparison classes. However, the effect size of 0.29 for the economics knowledge domain was 
positive and large enough to be considered substantively important according to WWC criteria (that 
is, at least 0.25 standard deviations). 


Review of the Report "Accommodations for English Language 
Learner Students: The Effect of Linguistic Modification of Math Test 
Item Sets” 

Citation: Sato, E., Rabinowitz, S., Gallagher, C., & Huang, C.-W. (2010). Accommodations for English 
language learner students: The effect of linguistic modification of math test item sets. (NCEE 2009-4079). 
Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. 

What is this study about? 

The study is a randomized controlled trial in which seventh- and eighth-grade students were 
randomly assigned to complete a set of 25 math questions delivered with either standard language or 
language that had undergone linguistic modification by the research team. 

The purpose of the study was to assess the effects of using linguistic modification as a way of removing 
language barriers for English language learners and non-English language learners (EL) struggling 
with reading. 

Nearly 3,000 students from 13 middle schools in five school districts in California were randomly 
assigned to complete traditional math assessments or math assessments that had undergone linguistic 
modification. Researchers then examined the results for three subgroups of students: Spanish-speaking 
EL, non-English language learners who were not proficient in English language arts (NEP), and 
non-English language learners who were proficient in English language arts (EP). Comparisons were 
made between students who took the modified test and those who took the non-modified test. 
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Features of Linguistic Modification 

The complexity of language used in test items may interfere with students’ abilities to demonstrate 
understanding of content, especially when students are stmggling with English. Linguistic modification 
is a test accommodation strategy aimed at removing language barriers. This strategy requires the 
creation of carefully constructed test items that are accessible to all students, regardless of language 
background, while still maintaining the integrity of the content being tested. This study focuses on 
the linguistic modification of math content that is typically presented on standardized math achievement 
tests. 

What did the study find? 

The study found a positive effect on math scores for students struggling with English who 
completed the linguistic modification item set relative to similar students who did not. The estimated six 
percentage-point gain on math achievement is statistically significant. The study found neither 
statistically significant nor substantively important differences for EP students who took the 
modified test, relative to those who did not. 
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Appendix E 

Sample and Weights for the REL Customer Survey 


The RELs serve 56 SEAs, including 50 states, the District of Columbia (DC), Puerto Rico (PR), 
Virgin Islands (VI), three territories: American Samoa (AS), the Commonwealth of the Northern 
Mariana Islands (MP), and Guam (GU). In the REL customer survey, all of these were included in 
the SEA stratum corresponding with their REL region. 

REL Pacific serves one state, Hawaii (HI); three territories, AS, GU, and MP; and also three nations 
in free association with the United States, the Federated States of Micronesia (FM), the Republic of 
the Marshall Islands (MH), and the Republic of Palau (PW). These three nations have neither state 
nor local educational agencies. However, since they are served by a REL, any users from these 
nations were eligible for the survey. The sample design covered the three nations in the SEA stratum 
for the Pacific region but combined them into one unit, which is referred to as the Pacific Nations. 
Another unique feature of REL Pacific is that all SEAs served by the REL have only one school 
district, so the SEAs and LEAs coincide. Therefore, there is no LEA stratum for the Pacific region. 

The study team used a two-stage design with LEAs as the primary sampling unit (PSU). The 
Common Core of Data (CCD) was used as the sampling frame. The CCD is produced annually by 
the National Center for Education Statistics (NCES). All LEAs were included in the sampling frame 
except supervisory union administrative centers, which do not have any local school districts; 
Department of Defense (DoD) dependents schools located overseas; and agencies without students 
in prekindergarten (PK) through grade 12. The frame consisted of all other types of LEAs, including 
LEAs designated as domestic DoD and Bureau of Indian Education (BIE). 

Since they were administrative units with REL users, supervisory unions were included regardless of 
whether they had students directly assigned to them as long as they supervised other districts that 
served students. Districts that were not supervisory unions and were serving no students were 
excluded. 
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Sample Design for the SEA Strata 


The following sampling scheme was used, depending on the SEA size, i.e., number of users (M): 


If M < 5, take all; 

If 5 < M < 10, take a simple random sample of 5; 

If 10 < M < 15, take a simple random sample of 7; 

If M > 15, take a simple random sample of 10. 

The sample selected as described above contained 346 users out of 569 in the SEA population. 
Table E-l shows the population and sample distribution by SEA. 


Table E-l. SEA user frame size, sample size, and sampling rate 


REL name 

SEA user 
frame size 

SEA user 
sample size 

Overall sampling 
rate (%) 

Pacific 

36 

17 

47.2 

Appalachia 

45 

28 

62.2 

Central 

67 

44 

65.7 

Mid-Atlantic 

52 

32 

61.5 

Midwest 

79 

47 

59.5 

Northeast and Islands 

72 

46 

63.9 

Northwest 

54 

33 

61.1 

Southeast 

66 

40 

60.6 

Southwest 

53 

33 

62.3 

West 

45 

26 

57.8 

Total 

569 

346 

60.8 


Sample Design for the LEA Strata 

For the LEA strata, users were reached using a two-stage cluster sample design, where a sample of 
LEAs was selected first, then users were selected from within the LEAs. To control the survey cost 
and enhance the survey efficiency, the study team planned to use a cluster design with a nearly equal 
probability for selection of users through the Probability-Proportional-to-Size (PPS) sampling 
method for LEA selection. 1 Since the number of users was not known when the LEAs were 
selected, the square root of student enrollment was used as the proxy size measure. 

1 The design effect, which is an indicator of sampling efficiency, consists of two main factors: the weighting factor and 
the clustering factor. A sample with a larger design effect is less efficient. The design effect increases as weights are 
more variable and/ or the average cluster size increases. Therefore, the two goals (controlling the survey cost and 
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The precision requirement of the survey was set at the REL level, since data were to be analyzed by 
REL. To determine the sample size to meet the REL-level precision, the following assumptions 
were made: (1) the target precision was set to be 3 percentage points for a population proportion of 
50 percent for a user characteristic — the half length of the 95 percent confidence interval was then 6 
percentage points; (2) the expected response rate was 80 percent; and (3) the design effect in the 
final nonresponse adjusted weights was two. 2 (4) The average number of users per LEA was seven. 

The needed sample sizes derived for each of the nine RELs based on the above assumption were 
250 LEAs and 1,750 users from the 250 LEAs. The total sample sizes for the nine RELs were then 
2,250 LEAs and 15,750 users. 

The study team selected an LEA sample of 2,250 LEAs based on the PPS method using the square 
root of LEA enrollment as the size measure and compiled lists of users for sampled LEAs. As the 
user lists were accumulated, it became clear that the number of users per LEA did not vary as much 
as expected, and the average number of users was smaller than anticipated (the unweighted average 
size was 4.7 whereas the weighted average was 3.3). In addition, the proxy size measure used for 
sampling (the square root of enrollment) was inaccurate, which if left untreated could cause large 
variation in the sampling weights, contrary to the original plan of a nearly self-weighting design (i.e., 
equal sampling weights). Moreover, it appeared that the target sample size of users would not be 
reached unless the LEA sample size was substantially increased. To remedy the problem, a simpler 
sample design was adopted that used a one-stage cluster design instead of two-stage design. 

The new design selected LEAs by simple random sampling and selected all users from the sampled 
LEAs. This design was an equal probability sampling of users. Because work had already begun on 
development of the user frame, the study team used the Keyfitz procedure (1951) to maximize the 
overlap between the old sample and the new sample of LEAs. 

In designing the sample, the study team originally planned to achieve an 80 percent response rate. 

To determine whether this was realistic, the data collection procedure was tested with a sample of 


increasing the sampling efficiency) are in conflict for a fixed sample size because the survey cost increases as the 
number of clusters (LEAs) increases (i.e., a larger number of user lists would need to be compiled). In contrast, 
sampling efficiency increases with the number of clusters in a sample of a given size. The evaluation team sought to 
choose a sample design that strikes a balance. An equal probability sample helps to enhance the sampling efficiency 
because it reduces the weighting factor in the design effect. 

2 This design effect was based on the assumption that seven users per LEA would be selected and that the intra-class 
correlation was 0.1. The evaluation team also assumed a moderate inflation (25%) in variance due to unequal weights 
that result from irregularities in sampling and nonresponse adjustment weighting. 
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1,027 users. The research team found that a 70 percent response rate could be achieved using the 
planned data collection procedures. Achieving an 80 percent response rate would require extensive 
follow-up with nonrespondents, more than originally planned. To free up the resources needed to 
achieve the target response rate without increasing the overall budget, the sample size was reduced. 
The final sample design was a one-stage cluster design that selected the LEA sample by simple 
random sampling and all users from selected LEAs without subsampling. The sample size of this 
final design is shown in Table E-2. 


Table E-2. The sample sizes for the revised LEA design by REL 


Frame size 

New LEA sample size 

Estimated average 
number of users 

Expected user 
sample size 

597 

116 

4.805 

556 

1,913 

213 

2.371 

505 

1,562 

164 

3.373 

554 

4,758 

211 

2.642 

556 

1,908 

174 

3.177 

553 

1,244 

186 

2.664 

496 

965 

123 

4.981 

610 

2,439 

182 

3.071 

559 

2,004 

147 

4.133 

608 

17,390 

1,516 

3.297 

4,998 


Results of Sample Selection and Data Collection 

Using the sample design described in the preceding sections, SEA and LEA samples were selected. 
The selected samples are summarized in Table E-3. 


Table E-3. Selected SEA and LEA sample sizes by REL 


REL 

State administrator 
sample size 

LEA 

sample size 

District administrator 
sample size 

Pacific 

17 

NA 

NA 

Appalachia 

28 

116 

534 

Central 

44 

213 

511 

Mid-Atlantic 

32 

164 

543 

Midwest 

47 

211 

546 

Northeast and Islands 

46 

174 

538 

Northwest 

33 

186 

474 

Southeast 

40 

123 

596 

Southwest 

33 

182 

566 

West 

26 

147 

526 

Total 

346 

1,516 

4,834 
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When the sample was fielded, positions in 12 LEAs had been eliminated, so the district 
administrator sample was reduced to 4,641, plus 181 district administrators found to be ineligible for 
the survey. The state administrator sample was 332, plus 14 state administrators found to be 
ineligible for the survey. 


Weighting 


Calculation of sampling weights was done in two steps. The first step was to weight the data by the 
inverse of the selection probability, and the second was to perform nonresponse adjustment. The 
resulting weights from these two steps were used to produce point estimates. To facilitate variance 
estimation, the study team created jackknife replicates and replicate weights. Each of these steps is 
explained in the sections that follow. 

The original selection probability provided the basis for the base weights. For the LEA sample, the 
base weights were all equal within each REL because LEAs were selected by an equal probability 
sampling method (i.e., simple random sampling), and there was no subsampling of users. Note that 
the selection probability of the LEA was the same as the user selection probability and so were their 
base weights. However, researchers subsequently adjusted the base weights for unit-level 
nonresponse. 


For nonresponse adjustment, weighting cells were created, and nonresponse adjustment was 
performed independently, cell by cell. The weighting cells were created using variables that are 
predictive of the response propensity. The study team used three variables to define the weighting 
cells for the LEA sample within each REL: (1) metro status (1 = Metro and 0 = Non-Metro); (2) 
type of LEA (regular and non-regular) 3 ; and (3) LEA size class (small, medium, and large) 4 . There 
were 12 possible cells, but if the number of respondents in a cell was less than 20, it was collapsed 


3 The CCD file from which the LEA sampling frame was created provided an LEA type variable with eight different 
types: 1 = Local school district that is not a component of a supervisory union; 2 = Local school district component 
of a supervisory union sharing a superintendent and administrative services with other local school districts; 3 = 
Supervisory union administrative center, or a county superintendent serving the same purpose; 4 = Regional education 
services agency, or a county superintendent serving the same purpose; 5 = State-operated institution charged, at least 
in part, with providing elementary and/ or secondary instruction or services to a special-needs population; 6 = 
Federally operated institution charged, at least in part, with providing elementary and/ or secondary instruction or 
services to a special-needs population; 7 = Agencies for which all associated schools are charter schools; 8 = Other 
education agencies that do not fit into the first seven categories. We defined the first three types (1, 2, and 3) as the 
regular districts and the remainder as the non-regular districts. 

4 The size class was defined based on district enrollment. Three equal classes were defined in terms of the number of 
districts, so the cut-points that defined the size classes varied from REL to REL. 
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into another cell with a similar response rate. The team also controlled the adjustment factor, which 
is the ratio of the sum of the base weights of all sample units in a cell to the sum of the base weights 
of the respondents in the cell. If this factor was greater than 2, the cell was collapsed into another 
cell so the factor did not exceed the limit. Collapsing of small cells and cells with overly large factors 
was done to control the variance inflation from the large variability in the adjusted weights. When 
the necessary collapsing was done, the adjustment factor was recalculated, and the nonresponse 
adjusted weight was computed as the product of the factor and the base weight. This adjusted 
weight was used in analysis. For the SEA sample, nonresponse adjustments were made within each 
REL, treating the whole REL as the weighting cell. 

To facilitate variance estimation, 40 replicate weights were developed for the SEA sample, with each 
user as the primary sampling unit (PSU); 60 replicate weights were developed for the LEA sample, 
with each LEA as the PSU. 5 The combined sample has 100 replicate weights. For REL-specific 
analyses, there are always 60 replicates for the LEA sample since each REL has more than 60 LEAs. 
However, for most RELs, there are fewer than 40 replicates for the SEA sample. 

Nevertheless, the total number of replicates is always greater than 60, which is usually adequate to 
maintain the stability of the variance estimate. 6 When the data is analyzed for the SEA stratum for 
each REL, the number of replicates is 40 or less, but the variance estimate for the SEA stratum is 
still stable because the finite population correction is small. 

To create the replicates, PSUs were first paired randomly to form variance strata within the sample 
type (SEA and LEA). Then the paired jackknife variance estimator was defined (JK2) (Westat, 

2002). Since the finite population correction factors were not negligible, especially for the SEA 
sample, the factors were developed for variance estimation. If the factors had not been used, the 
variance would have been over-estimated. 


5 The jackknife method defines replicates based on the PSU, which is the first-stage sampling unit. The element 
sampling method used for the SEA sample selects the ultimate sampling units (users in this case) directly, whereas the 
cluster sampling method selects the first-stage clusters (LEAs in this case) as PSU and then the ultimate sampling units 
from the selected PSUs. 

6 The number of replicates determines the degrees of freedom of the variance estimate, which in turn determines the 
stability of the variance estimate. In this case, they are roughly the same, that is, the number of degrees of freedom is 
the same as the number of replicates. The larger the number of degrees of freedom, the more stable the variance 
estimate becomes. A number of degrees of freedom such as 60, is generally considered adequate as the efficiency loss 
is less than 4 percent (see Korn and Graubard, 1999). 

7 The finite population correction is one minus the sampling rate. This correction factor is multiplied by the 
uncorrected variance estimate, and the corrected variance estimate becomes small if the factor is small. The overall 
finite population correction was 0.43 for all SEA strata. 
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A. YOUR EDUCATION RESEARCH AND TECHNICAL ASSISTANCE NEEDS 

To better meet the needs of education policymakers and practitioners, the National Center for Education 
Evaluation (NCEE), part of the U.S. Department of Education’s Institute for Education Sciences (IES), 
invites you to participate in this brief survey about the Regional Educational Laboratories (RELs) and your 
education research and technical assistance needs. 

As you answer the survey questions, please focus on the area or areas you are responsible for. We begin by 
asking about your education research and technical assistance needs. 

Al. Please indicate from the topics below your level of need for research and/ or technical assistance. If 
needed, select a response that balances your need for different types of research and technical assistance. 

(Select one response in each row.) 



High need for 

Moderate need 

Low or no 
need for 


research 

for research 

research 


and/or 

and/or 

and/or 


assistance 

assistance 

assistance 

a. Achievement Gaps 

□ 

□ 

□ 

b. Assessment (Formative or Summative) 

□ 

□ 

□ 

c. Behavior, Character Education, or Health 

□ 

□ 

□ 

d. College or Career Readiness 

□ 

□ 

□ 

e. Content Standards, Curriculum or Instruction in: 
el. Science, Technology, Engineering or Mathematics 

□ 

□ 

□ 

e2. Reading/Writing 

□ 

□ 

□ 

e3. Other (e.g.. Social Studies, Fine Arts) 

□ 

□ 

□ 

f. Dropout Prevention 

□ 

□ 

□ 

g. Early Childhood 

□ 

□ 

□ 

h. English Language Learners 

□ 

□ 

□ 

i. High School Reform 

□ 

□ 

□ 

j. Leadership 

□ 

□ 

□ 

k. Longitudinal Data Systems 

□ 

□ 

□ 

1. Parental Involvement 

□ 

□ 

□ 

m. Professional Development 

□ 

□ 

□ 

n. Rural Schools 

□ 

□ 

□ 

o. School Accountability 

□ 

□ 

□ 

p. School Choice 

□ 

□ 

□ 


i 


This appendix does not include the screen shots visible to respondents but rather the survey used to program the web 
survey. 
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High need for 

Moderate need 

Low or no 
need for 


research 

for research 

research 


and/or 

and/or 

and/or 


assistance 

assistance 

assistance 

q. School Finance 

□ 

□ 

□ 

r. Students with Disabilities 

□ 

□ 

□ 

s. Supplemental Education Services 

□ 

□ 

□ 

t. Support for Low Achieving Schools 

□ 

□ 

□ 

u. Teacher/Staff Evaluation 

□ 

□ 

□ 

v. Using Data for Decisions 

□ 

□ 

□ 

w. Other (Please specify) 

□ 

□ 

□ 


A2. To what extent do you rely on each of the following sources for education research or technical 
assistance? (Select one response in each row.) 


Sources 

a. The REL program nationwide 

b. If marked 1, 2, or 3 for (a) ask about: [REL Appalachia, 
CNA; McREL, etc.] 

c. U.S. Department of Education’s (ED) 
Comprehensive Centers 

d. Other federally funded technical assistance providers 

e. Other products or resources from ED including 
websites such as Doing What Works 

f. Professional associations (e.g., ASCD, Council of 
Chief State School Officers) 

g. Colleges and universities 

h. Consulting firms or private contractors 

i. Your counterparts at other LEAs or SEAs 

j. A technical assistance center supported by your state 

k. Education journals and publications 

l. Other (Please specify) 



Rely on 


Do not 

Rely on 

source to a 

Rely on 

rely on 

source to a 

moderate 

source to a 

source at 

great extent 

extent 

small extent 

all 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


A3. How easy is it for you to access education research and/or technical assistance when you need it? 

MARK ONE ANSWER 

□ Very easy 

□ Moderately easy 

□ Not at all easy 
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REL Customer Survey 


A4. Overall, how well do your sources for education research and/or technical assistance meet your needs. 
Would you say . . . 

MARK ONE ANSWER 

□ Very well 

□ Moderately well 

□ Not well 

The REL program consists of a network of laboratories that serve the educational needs of a designated region 
by providing access to research and technical assistance activities. 

We would like to find out about your experience with the REL program nationwide, including [NAME OF 
REGIONAL REL - ORGANIZATION, such as REL Appalachia at CNA] . 

Bl. How familiar are you with the REL program overall (e.g., Ask A REL, reports produced by the RELs, 
conferences or Bridge Events held by the RELs)? 

MARK ONE ANSWER 
D Very familiar 
D Somewhat familiar 
D A little familiar 

□ Not f a miliar at all — > SKIP TO END 

B2. Other than reports, which services provided by [NAME OF REGIONAL REL - ORGANIZATION] 
have you used in the past 12 months? 

MARK ALL THAT APPLY, THEN, IF ANY ARE MARKED, SKIP TO B4 

□ A live or virtual event (e.g., Bridge Event, Webinar) 

□ Technical assistance 

EH Responses to data or research requests via email or phone (e.g. Ask A REL) 

□ Information on the REL’s website 

□ Some other service ( Please specify) 

D I haven’t used any services provided by the REL in my region in the past 12 months 

D I don’t know if services I used in the past 12 months were provided by the REL in my region. : 

SKIP TO B5 

B3. Why haven’t you used the services provided by [NAME OF REGIONAL REL - ORGANIZATION] 
in the past 12 months? 

MARK ALL THAT APPLY, THEN SKIP TO B5 

D I had no need for the REL’s resources in the past 12 months. 

□ My needs were met elsewhere. 

D The REL in my region does not have a good reputation. 

D There is not a good match between my current needs and the REL’s resources. 

□ I didn’t know what services were available through the REL. 

□ Some other reason (Please specify) 
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REL Customer Survey 

B4. What type of contact did you have with [NAME OF REGIONAL REL - ORGANIZATION]? 

MARK ALL THAT APPLY 

D I contacted a reference desk for help or used the Ask A REL link on the web site. 

□ I attended a REL sponsored conference, training, or workshop 

□ A REL representative was present at a meeting or workshop I attended. 

□ My organization and/ or I contacted the REL for research or other assistance. 

D I forwarded a request that came to me or my organization to the REL. 

D Some other means (Phase specify) 

B5. Have you ever read any reports produced by one of the RELs nationwide, including [NAME OF 
REGIONAL REL - ORGANIZATION] ? 

□ Yes 

Ed No — >If responded No to B5 and checked response 6 for B2 SKIP TO END 

Ed Don’t know — > If responded Don’t know to B5 and checked response 6 for B2 SKIP TO END 

B6. In the last 12 months, how well did the REL program nationwide, including [NAME OF 
REGIONAL REL - ORGANIZATION] meet your education research and technical 
assistance needs? If needed, select a response that balances your need for different types of research 
and technical assistance. (Select one response in each row.) 

[WEB INSTRUCTIONS: FOR EACH TOPIC AREA MARKED 1 OR 2 IN Al, ASK:] 

Did not 
receive 
assistance 


The REL met my research and technical assistance 
needs... 

Very well 

Moderately 

well 

Not at all 
well 

from the 
REL 

a. Achievement Gaps 

□ 

□ 

□ 

□ 

b. Assessment (Formative or Summative) 

□ 

□ 

□ 

□ 

c. Behavior, Character Education, or Health 

□ 

□ 

□ 

□ 

d. College or Career Readiness 

□ 

□ 

□ 

□ 

e. Content Standards, Curriculum or Instruction in: 
el. Science, Technology, Engineering or Mathematics 

□ 

□ 

□ 

□ 

e2. Reading/Writing 

□ 

□ 

□ 

□ 

e3. Other (e.g., Social Studies, Fine Arts) 

□ 

□ 

□ 

□ 

f. Dropout Prevention 

□ 

□ 

□ 

□ 

g. Early Childhood 

□ 

□ 

□ 

□ 

h. English Language Learners 

□ 

□ 

□ 

□ 

i. High School Reform 

□ 

□ 

□ 

□ 

j. Leadership 

□ 

□ 

□ 

□ 

k. Longitudinal Data Systems 

□ 

□ 

□ 

□ 

1. Parental Involvement 

□ 

□ 

□ 

□ 

m. Professional Development 

□ 

□ 

□ 

□ 

n. Rural Schools 

□ 

□ 

□ 

□ 

o. School Accountability 

□ 

□ 

□ 

□ 
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The REL met my research and technical assistance 
needs... 

p. School Choice 

q. School Finance 

r. Students with Disabilities 

s. Supplemental Education Services 

t. Teacher/Staff Evaluation 

u. Support for Low Achieving Schools 

v. Using Data for Decisions 

w. Other (Please specify) 


Did not 
receive 
assistance 



Moderately 

Not at all 

from the 

Very well 

well 

well 

REL 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


B7. Please consider all your experiences in the past 12 months with the REL program nationwide, 

including [NAME OF REGIONAL REL - ORGANIZATION] (i.e., any reports from the RELs that 
you have read, conferences or Bridge Events that you have attended, and other services you or your 
organization received from the RELs). The work of the RELs... (Select one response in each row.) 



To a high 

To a moderate 

To a low 

Not able to 


degree 

degree 

degree 

judge 

a. Addressed an important need or 





problem that you or your organization 
face 

□ 

□ 

□ 

□ 

b. Addressed a topic in a timely manner 

□ 

□ 

□ 

□ 

for you or your organization 





c. Contributed new information on the 

□ 

□ 

□ 

□ 

topic being addressed 

d. Provided information that could be 





used to inform decisions about 
policies, programs, or practices 

□ 

□ 

□ 

□ 

e. Was convenient to access 

□ 

□ 

□ 

□ 

f. Was presented in a way that was easy to 

□ 

□ 

□ 

□ 

understand 

g. Was presented in a way that was easy 

□ 

□ 

□ 

□ 

to use 

h. Provided information that you or your 

□ 

□ 

□ 

□ 

organization will use again 

B8. Overall, how satisfied were you with the work of the REL program nationwide including [NAME OF 


REGIONAL REL - ORGANIZATION] ? 

MARK ONE ANSWER 

□ Very satisfied 

□ Somewhat satisfied 

□ Not at all satisfied [If not at all satisfied, a pop-zip text box uill appear with the question 'Please explain briefly why yon 
are not satisfied with the RELs. ”] 

Thank you very much for your time. 
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Appendix G 

Technical Working Group Members 


Judy Arter 

Educational Testing Service 
Rolf Blank 

Council of Chief State School Officers 
Gregg Jackson 

George Washington University 

Conrad Katzenmeyer 
University of Central Florida 

Larry Ludlow 
Boston College 

Larry Orr 

Independent Consultant 
Colleen Serement 

Maryland State Department of Education 
Deb Sigman 

California Department of Education 
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